Skip to content

5. Flow parameters

Replace the fixed basket threshold with a typed flow parameter. Keep the aggregation from chapter 4. The complete script contains the complete pipeline.

minimum = ff.add_flow_parameter(enriched, ff.Parameter("min_quantity", default=8, type="integer"))
bulk = enriched.filter(ff.col("quantity") >= minimum)

add_flow_parameter() declares the parameter on the graph and returns the Parameter you pass in. It accepts either a FlowFrame or a FlowGraph: passing enriched.flow_graph instead of enriched declares the same parameter on the same graph.

A Python variable exists in this script. A Parameter declares a named value stored with the flow, so a later run can override it. default=8 is its initial value; type="integer" checks values supplied through the parameter helpers.

Declare the parameter on enriched before building the filter. The filter retains the parameter reference, rather than storing only the number eight.

Change a run

print(report.collect())

ff.set_flow_parameter(report, minimum, 10)
report.flow_graph.run_graph()
print(report.flow_graph.get_node(report.node_id).get_resulting_data().collect())

set_flow_parameter() updates the graph's stored value. For this ordinary lazy frame, collect() still evaluates the plan built earlier. Run the graph explicitly, then read the report node's result to observe the new value.

A parameter is a value

Use it in a comparison, not as a column name. Declaring min_quantity does not create a data column. minimum.default remains eight even after changing the graph's copy with set_flow_parameter().

Check it

The script prints the report at the default of eight, then at ten:

Output
shape: (5, 5)
┌───────────┬────────┬──────────┬─────────┬───────────┐
│ city      ┆ orders ┆ revenue  ┆ target  ┆ shortfall │
│ ---       ┆ ---    ┆ ---      ┆ ---     ┆ ---       │
│ str       ┆ u32    ┆ f64      ┆ f64     ┆ f64       │
╞═══════════╪════════╪══════════╪═════════╪═══════════╡
│ Bago      ┆ 59     ┆ 28592.7  ┆ 30000.0 ┆ 1407.3    │
│ Mandalay  ┆ 73     ┆ 36937.24 ┆ 35000.0 ┆ -1937.24  │
│ Naypyitaw ┆ 51     ┆ 23733.16 ┆ 25000.0 ┆ 1266.84   │
│ Taunggyi  ┆ 63     ┆ 32710.95 ┆ 30000.0 ┆ -2710.95  │
│ Yangon    ┆ 68     ┆ 34096.0  ┆ 35000.0 ┆ 904.0     │
└───────────┴────────┴──────────┴─────────┴───────────┘
shape: (5, 5)
┌───────────┬────────┬─────────┬─────────┬───────────┐
│ city      ┆ orders ┆ revenue ┆ target  ┆ shortfall │
│ ---       ┆ ---    ┆ ---     ┆ ---     ┆ ---       │
│ str       ┆ u32    ┆ f64     ┆ f64     ┆ f64       │
╞═══════════╪════════╪═════════╪═════════╪═══════════╡
│ Bago      ┆ 14     ┆ 7334.9  ┆ 30000.0 ┆ 22665.1   │
│ Mandalay  ┆ 31     ┆ 16608.6 ┆ 35000.0 ┆ 18391.4   │
│ Naypyitaw ┆ 20     ┆ 10422.1 ┆ 25000.0 ┆ 14577.9   │
│ Taunggyi  ┆ 18     ┆ 9752.6  ┆ 30000.0 ┆ 20247.4   │
│ Yangon    ┆ 24     ┆ 15643.6 ┆ 35000.0 ┆ 19356.4   │
└───────────┴────────┴─────────┴─────────┴───────────┘

The first report matches chapter 4: its orders column sums to 314 invoices. At ten the same five cities keep 107, and every city falls below its target.

Complete script

Show the complete runnable script
tutorial_05.py
"""Make the basket threshold a flow parameter and change it between runs."""

import flowfile as ff

# Chapters 1-3: read, clean, and enrich the invoices.
SALES = "https://raw.githubusercontent.com/edwardvaneechoud/flowfile/main/data/templates/supermarket_sales.csv"
sales = ff.read_csv(SALES, description="Supermarket invoices")

valid = sales.unique().filter(flowfile_formula="[quantity] > 0 and [unit_price] > 0")
orders = valid.with_columns(flowfile_formulas=["[unit_price] * [quantity]"], output_column_names=["revenue"])

members = orders.filter(ff.col("customer_type") == "Member")
regulars = orders.filter(ff.col("customer_type") == "Normal")
combined = ff.concat([members, regulars], how="diagonal_relaxed")

targets = ff.from_dict(
    {
        "city": ["Bago", "Mandalay", "Naypyitaw", "Taunggyi", "Yangon"],
        "target": [30000.0, 35000.0, 25000.0, 30000.0, 35000.0],
    }
)
enriched = combined.join(targets, on="city", how="left")

# Chapter 5: the basket threshold becomes a flow parameter.
minimum = ff.add_flow_parameter(enriched, ff.Parameter("min_quantity", default=8, type="integer"))
bulk = enriched.filter(ff.col("quantity") >= minimum)

# Chapter 4: one report row per city.
summary = bulk.group_by("city").agg(
    ff.col("invoice_id").count().alias("orders"),
    ff.col("revenue").sum().alias("revenue"),
    ff.col("target").first().alias("target"),
)
report = summary.with_columns((ff.col("target") - ff.col("revenue")).alias("shortfall")).sort("city")

print(report.collect())

ff.set_flow_parameter(report, minimum, 10)
report.flow_graph.run_graph()
print(report.flow_graph.get_node(report.node_id).get_resulting_data().collect())

Download tutorial_05.py

Recap

The saved report now has a typed input instead of a fixed threshold. Next, use the resulting shortfalls to choose a review or ready route for the whole report.

Next: Branching with Gate