2. Clean and shape
Continue the invoice pipeline with positive-quantity and positive-price checks. Express the revenue rule as a Flowfile formula, then compare it with the expression form. The complete script repeats the reader from chapter 1.
valid = sales.unique().filter(flowfile_formula="[quantity] > 0 and [unit_price] > 0")
orders = valid.with_columns(flowfile_formulas=["[unit_price] * [quantity]"], output_column_names=["revenue"])
filter() keeps matching rows. with_columns() adds or replaces columns without reducing the row count. These sample rows already pass the positive-value check; deduplication is what removes the 30 extra rows.
Formulas and expressions
expression_orders = valid.with_columns((ff.col("unit_price") * ff.col("quantity")).alias("revenue"))
scaled = orders.with_columns(
(ff.col("revenue") / 1000).alias("revenue"),
(ff.col("revenue") * 0.05).alias("expected_income"),
)
The multiplication expression becomes an editable Formula node too. Flowfile uses native nodes when it can represent the operation with their settings. Unsupported expression forms fall back to Polars code (polars_code).
The scaled comparison deliberately replaces revenue while another expression reads it. Both expressions read the original input to that call: expected income is still 5% of the unscaled revenue. Keeping those semantics requires a Polars code node. Continue the pipeline with orders; scaled is only this comparison.
A new column needs a second expression call
Within one expression-based with_columns() call, expressions do not read one another's new values. Chain a second call when a calculation depends on the column you just added. In contrast, entries passed through flowfile_formulas= evaluate in sequence and can reference earlier entries.
Check it
orders.flow_graph.print_tree()
print(scaled.select("invoice_id", "revenue", "expected_income").sort("invoice_id").head(3).collect())
The script prints:
● Read data (1) Supermarket invoices
│
● Drop duplicates (2)
│
● Filter data (3)
├─┐
│ ● Formula (8)
● Formula (6)
│
● Polars code (9) Add columns: revenue, expected_income
shape: (3, 3)
┌────────────┬─────────┬─────────────────┐
│ invoice_id ┆ revenue ┆ expected_income │
│ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 │
╞════════════╪═════════╪═════════════════╡
│ INV-00001 ┆ 0.04041 ┆ 2.0205 │
│ INV-00002 ┆ 0.6389 ┆ 31.945 │
│ INV-00003 ┆ 0.39159 ┆ 19.5795 │
└────────────┴─────────┴─────────────────┘
After Filter data the graph branches. The Formula on the right is expression_orders, which nothing reads. The Formula on the left is orders, and the Polars code node below it is scaled. For INV-00001, scaled revenue is 0.04041 (40.41 / 1000) and expected income is 2.0205 (5% of 40.41).
Complete script
Show the complete runnable script
"""Clean the invoices with editable formulas and compare them with expressions."""
import flowfile as ff
# Chapter 1: read the invoices.
SALES = "https://raw.githubusercontent.com/edwardvaneechoud/flowfile/main/data/templates/supermarket_sales.csv"
sales = ff.read_csv(SALES, description="Supermarket invoices")
# Chapter 2: keep valid invoices and add revenue.
valid = sales.unique().filter(flowfile_formula="[quantity] > 0 and [unit_price] > 0")
orders = valid.with_columns(flowfile_formulas=["[unit_price] * [quantity]"], output_column_names=["revenue"])
expression_orders = valid.with_columns((ff.col("unit_price") * ff.col("quantity")).alias("revenue"))
scaled = orders.with_columns(
(ff.col("revenue") / 1000).alias("revenue"),
(ff.col("revenue") * 0.05).alias("expected_income"),
)
orders.flow_graph.print_tree()
print(scaled.select("invoice_id", "revenue", "expected_income").sort("invoice_id").head(3).collect())
Recap
You can choose a readable formula without giving up expressions. Node selection depends on what the operation means, including which version of a column it reads. Next, combine invoice segments and attach city targets.
Next: Combine