Skip to content

8. Reusable flows

Extract the filter, aggregation, and shortfall calculation into a child flow. The parent still reads, cleans, enriches, and routes the report. The complete script is runnable on its own.

incoming = ff.FlowInput("invoices", schema=enriched.schema)
child_minimum = ff.add_flow_parameter(incoming, ff.Parameter("min_quantity", default=8, type="integer"))
child_summary = (
    incoming.filter(ff.col("quantity") >= child_minimum)
    .group_by("city")
    .agg(
        ff.col("invoice_id").count().alias("orders"),
        ff.col("revenue").sum().alias("revenue"),
        ff.col("target").first().alias("target"),
    )
)
child_report = child_summary.with_columns((ff.col("target") - ff.col("revenue")).alias("shortfall")).sort("city")
child_report.to_flow_output("city_report")
report_flow = ff.register_flow(child_report, name="Tutorial city summary")

This replaces the inline report calculation from chapters 4–5. FlowInput("invoices", schema=...) declares a named input with columns but no rows. to_flow_output("city_report") names the result the parent can retrieve. register_flow() writes the child YAML and registers it in the catalog immediately.

Feed the child

minimum = ff.add_flow_parameter(enriched, ff.Parameter("min_quantity", default=8, type="integer"))
call = ff.RunFlow(report_flow, invoices=enriched, params={"min_quantity": minimum})
report = call["city_report"]

invoices=enriched binds a frame to the child's input slot. params= passes the parent's threshold to the child's parameter. The names serve different purposes: one carries rows, the other carries a value.

The Gate from chapter 6 now reads report. A RunFlow output is deferred: building it does not run the child; collecting it runs its lineage. The complete script leaves out the chapter 7 writers; chapter 10 puts the whole pipeline together.

Compare thresholds with iterate

scenarios = ff.from_dict({"minimum": [1, 8, 10]})
comparison = ff.RunFlow(
    report_flow,
    invoices=enriched,
    params={"min_quantity": ff.col("minimum")},
    param_frame=scenarios,
    iterate=True,
)
per_threshold = (
    comparison["city_report"]
    .group_by("param_min_quantity")
    .agg(ff.col("city").count().alias("cities"), ff.col("orders").sum().alias("invoices"))
    .sort("param_min_quantity")
)

param_frame supplies one threshold per row. ff.col("minimum") binds that column to the child's min_quantity. With iterate=True, the child runs three times and returns the combined outputs with param_min_quantity and run_index metadata. The script then counts cities and sums invoices per threshold.

Iteration does not partition the invoices

Every run receives the same enriched frame. Only the bound parameter changes. The child filter selects the qualifying invoices; iterate=True itself does not split them into groups.

Check it

child_report.flow_graph.print_tree()
print()
routed.flow_graph.print_tree()
print(report.collect())
print(per_threshold.collect())

The script prints:

Output
● Flow Input (20)
│
● Filter data (21)
│
● Group by (22)
│
● Formula (24)
│
● Sort data (25)
│
● Flow Output (26)

● Read data (11)  Supermarket invoices
│
● Drop duplicates (12)
│
● Filter data (13)
│
● Formula (14)
├─┐
● │ Filter data (15)
│ ● Filter data (16)
├─┘
● Union data (17)
│ ● Manual input (18)
├─┘
● Join (19)
│
● Run Flow (27)
│
● Gate (28)
├─┐
● │ Formula (30)  [then]
│ ● Formula (32)  [else]
├─┘
● Union data (33)
shape: (5, 5)
┌───────────┬────────┬──────────┬─────────┬───────────┐
│ city      ┆ orders ┆ revenue  ┆ target  ┆ shortfall │
│ ---       ┆ ---    ┆ ---      ┆ ---     ┆ ---       │
│ str       ┆ u32    ┆ f64      ┆ f64     ┆ f64       │
╞═══════════╪════════╪══════════╪═════════╪═══════════╡
│ Bago      ┆ 59     ┆ 28592.7  ┆ 30000.0 ┆ 1407.3    │
│ Mandalay  ┆ 73     ┆ 36937.24 ┆ 35000.0 ┆ -1937.24  │
│ Naypyitaw ┆ 51     ┆ 23733.16 ┆ 25000.0 ┆ 1266.84   │
│ Taunggyi  ┆ 63     ┆ 32710.95 ┆ 30000.0 ┆ -2710.95  │
│ Yangon    ┆ 68     ┆ 34096.0  ┆ 35000.0 ┆ 904.0     │
└───────────┴────────┴──────────┴─────────┴───────────┘
shape: (3, 3)
┌────────────────────┬────────┬──────────┐
│ param_min_quantity ┆ cities ┆ invoices │
│ ---                ┆ ---    ┆ ---      │
│ i64                ┆ u32    ┆ u32      │
╞════════════════════╪════════╪══════════╡
│ 1                  ┆ 5      ┆ 1000     │
│ 8                  ┆ 5      ┆ 314      │
│ 10                 ┆ 5      ┆ 107      │
└────────────────────┴────────┴──────────┘

The first graph is the child: Flow Input and Flow Output around the filter and aggregation. In the parent, a single Run Flow node takes their place between Join and Gate. The child's report matches chapter 4, and the threshold comparison counts 1,000, 314, and 107 qualifying invoices across the same five cities.

Complete script

Show the complete runnable script
tutorial_08.py
"""Extract the city report into a reusable child flow and compare thresholds."""

import flowfile as ff

# Chapters 1-3: read, clean, and enrich the invoices.
SALES = "https://raw.githubusercontent.com/edwardvaneechoud/flowfile/main/data/templates/supermarket_sales.csv"
sales = ff.read_csv(SALES, description="Supermarket invoices")

valid = sales.unique().filter(flowfile_formula="[quantity] > 0 and [unit_price] > 0")
orders = valid.with_columns(flowfile_formulas=["[unit_price] * [quantity]"], output_column_names=["revenue"])

members = orders.filter(ff.col("customer_type") == "Member")
regulars = orders.filter(ff.col("customer_type") == "Normal")
combined = ff.concat([members, regulars], how="diagonal_relaxed")

targets = ff.from_dict(
    {
        "city": ["Bago", "Mandalay", "Naypyitaw", "Taunggyi", "Yangon"],
        "target": [30000.0, 35000.0, 25000.0, 30000.0, 35000.0],
    }
)
enriched = combined.join(targets, on="city", how="left")

# Chapter 8: the report calculation from chapters 4-5 becomes a child flow.
incoming = ff.FlowInput("invoices", schema=enriched.schema)
child_minimum = ff.add_flow_parameter(incoming, ff.Parameter("min_quantity", default=8, type="integer"))
child_summary = (
    incoming.filter(ff.col("quantity") >= child_minimum)
    .group_by("city")
    .agg(
        ff.col("invoice_id").count().alias("orders"),
        ff.col("revenue").sum().alias("revenue"),
        ff.col("target").first().alias("target"),
    )
)
child_report = child_summary.with_columns((ff.col("target") - ff.col("revenue")).alias("shortfall")).sort("city")
child_report.to_flow_output("city_report")
report_flow = ff.register_flow(child_report, name="Tutorial city summary")

minimum = ff.add_flow_parameter(enriched, ff.Parameter("min_quantity", default=8, type="integer"))
call = ff.RunFlow(report_flow, invoices=enriched, params={"min_quantity": minimum})
report = call["city_report"]

# Chapter 6: the Gate now reads the child flow's report.
gate = ff.Gate(report, ff.col("shortfall") > 0)
review = gate.then.with_columns(ff.lit("review").alias("route"))
ready = gate.otherwise.with_columns(ff.lit("ready").alias("route"))
routed = ff.concat([review, ready], how="diagonal_relaxed")

scenarios = ff.from_dict({"minimum": [1, 8, 10]})
comparison = ff.RunFlow(
    report_flow,
    invoices=enriched,
    params={"min_quantity": ff.col("minimum")},
    param_frame=scenarios,
    iterate=True,
)
per_threshold = (
    comparison["city_report"]
    .group_by("param_min_quantity")
    .agg(ff.col("city").count().alias("cities"), ff.col("orders").sum().alias("invoices"))
    .sort("param_min_quantity")
)

child_report.flow_graph.print_tree()
print()
routed.flow_graph.print_tree()
print(report.collect())
print(per_threshold.collect())

Download tutorial_08.py

Recap

The report has a reusable interface: invoices in, one threshold, city report out. Next, add a comparable shortfall percentage using SQL and inspect equivalent custom-logic options.

Next: Custom logic