Skip to content

7. Write results

Persist the routed city report as files and a catalog table, with an optional cloud destination. The complete script includes the complete pipeline through the Gate.

Write files

output_dir = Path("city_report").resolve()
output_dir.mkdir(exist_ok=True)
routed.write_csv(output_dir / "city_report.csv")
routed.write_parquet(output_dir / "city_report.parquet")
routed.flow_graph.run_graph()

The script writes into a city_report folder in the directory you run it from. These writers sit below a Gate, so placing them does not write files. run_graph() executes the graph, including the writers. Collecting routed alone would not execute its downstream sinks.

Publish a catalog table

reports = ff.CatalogReference("tutorial_sales", auto_create=True).schema("reports", auto_create=True)
reports.write_table(routed, "city_report", write_mode="overwrite").collect()

The references name tutorial_sales.reports.city_report. auto_create=True creates missing catalog and schema namespaces. overwrite replaces the prior report on reruns. Collecting the writer's frame runs its lineage and performs this write.

Use a stored cloud connection

A cloud writer takes a stored connection by name. This example writes an aggregate to S3 and reads it back:

income_by_city = (
    ff.read_csv("data/templates/supermarket_sales.csv")
    .group_by("city")
    .agg(ff.col("gross_income").sum().alias("total_income"))
)

income_by_city.write_parquet_to_cloud_storage(s3_path, connection_name="analytics-s3")

reloaded = ff.scan_parquet_from_cloud_storage(s3_path, connection_name="analytics-s3").collect()

routed.write_parquet_to_cloud_storage(path, connection_name="analytics-s3") places the same writer below the report. Configure the connection through Cloud Connection Management. Connection credentials belong to that connection; the writer stores its name.

External-service checks

The docs test suite runs this example against MinIO. The chapter script leaves the cloud step out because it needs a stored connection and a writable bucket.

Check it

routed.flow_graph.print_tree()
print(ff.read_parquet(str(output_dir / "city_report.parquet")).collect())
print(reports.read_table("city_report").sort("city").collect())

The script prints:

Output
● Read data (11)  Supermarket invoices
│
● Drop duplicates (12)
│
● Filter data (13)
│
● Formula (14)
├─┐
● │ Filter data (15)
│ ● Filter data (16)
├─┘
● Union data (17)
│ ● Manual input (18)
├─┘
● Join (19)
│
● Filter data (20)
│
● Group by (21)
│
● Formula (23)
│
● Sort data (24)
│
● Gate (25)
├─┐
● │ Formula (27)  [then]
│ ● Formula (29)  [else]
├─┘
● Union data (30)
├─┬─┐
● │ │ Write data (31)
  ● │ Write data (32)
    ● Write to Catalog (33)
shape: (5, 6)
┌───────────┬────────┬──────────┬─────────┬───────────┬────────┐
│ city      ┆ orders ┆ revenue  ┆ target  ┆ shortfall ┆ route  │
│ ---       ┆ ---    ┆ ---      ┆ ---     ┆ ---       ┆ ---    │
│ str       ┆ u32    ┆ f64      ┆ f64     ┆ f64       ┆ str    │
╞═══════════╪════════╪══════════╪═════════╪═══════════╪════════╡
│ Bago      ┆ 59     ┆ 28592.7  ┆ 30000.0 ┆ 1407.3    ┆ review │
│ Mandalay  ┆ 73     ┆ 36937.24 ┆ 35000.0 ┆ -1937.24  ┆ review │
│ Naypyitaw ┆ 51     ┆ 23733.16 ┆ 25000.0 ┆ 1266.84   ┆ review │
│ Taunggyi  ┆ 63     ┆ 32710.95 ┆ 30000.0 ┆ -2710.95  ┆ review │
│ Yangon    ┆ 68     ┆ 34096.0  ┆ 35000.0 ┆ 904.0     ┆ review │
└───────────┴────────┴──────────┴─────────┴───────────┴────────┘
shape: (5, 6)
┌───────────┬────────┬──────────┬─────────┬───────────┬────────┐
│ city      ┆ orders ┆ revenue  ┆ target  ┆ shortfall ┆ route  │
│ ---       ┆ ---    ┆ ---      ┆ ---     ┆ ---       ┆ ---    │
│ str       ┆ i32    ┆ f64      ┆ f64     ┆ f64       ┆ str    │
╞═══════════╪════════╪══════════╪═════════╪═══════════╪════════╡
│ Bago      ┆ 59     ┆ 28592.7  ┆ 30000.0 ┆ 1407.3    ┆ review │
│ Mandalay  ┆ 73     ┆ 36937.24 ┆ 35000.0 ┆ -1937.24  ┆ review │
│ Naypyitaw ┆ 51     ┆ 23733.16 ┆ 25000.0 ┆ 1266.84   ┆ review │
│ Taunggyi  ┆ 63     ┆ 32710.95 ┆ 30000.0 ┆ -2710.95  ┆ review │
│ Yangon    ┆ 68     ┆ 34096.0  ┆ 35000.0 ┆ 904.0     ┆ review │
└───────────┴────────┴──────────┴─────────┴───────────┴────────┘

Three writers now branch off the final Union data: two Write data nodes for the CSV and Parquet files and Write to Catalog for the table. The Parquet file and the catalog table hold the same five review rows, with revenue and shortfall from chapter 4. orders comes back from the catalog as i32: catalog tables are stored in Delta format, which has no unsigned integer types.

Complete script

Show the complete runnable script
tutorial_07.py
"""Write the routed city report to files and a catalog table."""

from pathlib import Path

import flowfile as ff

# Chapters 1-3: read, clean, and enrich the invoices.
SALES = "https://raw.githubusercontent.com/edwardvaneechoud/flowfile/main/data/templates/supermarket_sales.csv"
sales = ff.read_csv(SALES, description="Supermarket invoices")

valid = sales.unique().filter(flowfile_formula="[quantity] > 0 and [unit_price] > 0")
orders = valid.with_columns(flowfile_formulas=["[unit_price] * [quantity]"], output_column_names=["revenue"])

members = orders.filter(ff.col("customer_type") == "Member")
regulars = orders.filter(ff.col("customer_type") == "Normal")
combined = ff.concat([members, regulars], how="diagonal_relaxed")

targets = ff.from_dict(
    {
        "city": ["Bago", "Mandalay", "Naypyitaw", "Taunggyi", "Yangon"],
        "target": [30000.0, 35000.0, 25000.0, 30000.0, 35000.0],
    }
)
enriched = combined.join(targets, on="city", how="left")

# Chapters 4-6: the parameterized city report, routed by a Gate.
minimum = ff.add_flow_parameter(enriched, ff.Parameter("min_quantity", default=8, type="integer"))
bulk = enriched.filter(ff.col("quantity") >= minimum)
summary = bulk.group_by("city").agg(
    ff.col("invoice_id").count().alias("orders"),
    ff.col("revenue").sum().alias("revenue"),
    ff.col("target").first().alias("target"),
)
report = summary.with_columns((ff.col("target") - ff.col("revenue")).alias("shortfall")).sort("city")

gate = ff.Gate(report, ff.col("shortfall") > 0)
review = gate.then.with_columns(ff.lit("review").alias("route"))
ready = gate.otherwise.with_columns(ff.lit("ready").alias("route"))
routed = ff.concat([review, ready], how="diagonal_relaxed")

# Chapter 7: write the routed report.
output_dir = Path("city_report").resolve()
output_dir.mkdir(exist_ok=True)
routed.write_csv(output_dir / "city_report.csv")
routed.write_parquet(output_dir / "city_report.parquet")
routed.flow_graph.run_graph()

reports = ff.CatalogReference("tutorial_sales", auto_create=True).schema("reports", auto_create=True)
reports.write_table(routed, "city_report", write_mode="overwrite").collect()

routed.flow_graph.print_tree()
print(ff.read_parquet(str(output_dir / "city_report.parquet")).collect())
print(reports.read_table("city_report").sort("city").collect())

Download tutorial_07.py

Recap

The report now has durable outputs and explicit write timing. Next, extract the report calculation so other flows can supply invoices and choose their own threshold.

Next: Reusable flows