1. First steps
Build a city sales report from supermarket invoices. This first step reads the public sample, removes exact duplicates, and calculates revenue; later chapters add targets, review routing, and scheduled runs.
import flowfile as ff
SALES = "https://raw.githubusercontent.com/edwardvaneechoud/flowfile/main/data/templates/supermarket_sales.csv"
sales = ff.read_csv(SALES, description="Supermarket invoices")
Use the environment from the Quick Start. The CSV URL works without a repository checkout. Complete runnable script.
You can pass an existing graph
In a standalone script, read_csv() creates a graph when you omit flow_graph. To add the reader to a graph you already have, pass it by keyword:
graph = ff.create_flow_graph()
sales_on_graph = ff.read_csv(SALES, flow_graph=graph)
The returned frame points to that same graph through sales_on_graph.flow_graph. You can also retrieve an automatically created graph from sales.flow_graph and pass it to another reader. This is an alternative setup; the walkthrough below continues with sales from the first example.
Build the transformation
orders = sales.unique().with_columns((ff.col("unit_price") * ff.col("quantity")).alias("revenue"))
preview = orders.select("invoice_id", "city", "revenue").sort("invoice_id")
unique() removes identical rows. with_columns() adds revenue as unit price × quantity. preview is a FlowFrame: it holds a lazy data plan and a reference to the graph built by these calls.
Check it
preview.flow_graph.print_tree()
print(f"{sales.collect().height} rows read")
print(preview.collect())
The script prints:
● Read data (1) Supermarket invoices
│
● Drop duplicates (2)
│
● Formula (4)
│
● Select data (5)
│
● Sort data (6)
1030 rows read
shape: (1_000, 3)
┌────────────┬───────────┬─────────┐
│ invoice_id ┆ city ┆ revenue │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 │
╞════════════╪═══════════╪═════════╡
│ INV-00001 ┆ Mandalay ┆ 40.41 │
│ INV-00002 ┆ Taunggyi ┆ 638.9 │
│ INV-00003 ┆ Bago ┆ 391.59 │
│ INV-00004 ┆ Naypyitaw ┆ 173.16 │
│ INV-00005 ┆ Bago ┆ 448.96 │
│ … ┆ … ┆ … │
│ INV-00996 ┆ Bago ┆ 514.71 │
│ INV-00997 ┆ Taunggyi ┆ 536.1 │
│ INV-00998 ┆ Bago ┆ 288.33 │
│ INV-00999 ┆ Taunggyi ┆ 846.9 │
│ INV-01000 ┆ Yangon ┆ 451.2 │
└────────────┴───────────┴─────────┘
print_tree() draws the graph without collecting any data: one node per line, top to bottom, named as on the canvas and followed by its node id. unique() became Drop duplicates and with_columns() a Formula. Ids are assigned as nodes are created and can skip numbers. Later graphs branch and merge; lines on the left show where.
collect() runs the plan and returns a Polars DataFrame: the file has 1,030 rows and 1,000 unique invoices. preview.flow_graph still holds the operations for the Designer.
Lazy does not mean no I/O
Reading this URL can fetch the file and inspect its schema while the graph is built. Laziness here means the transformation result is not materialized until collection. Writers and kernel nodes have their own execution rules, introduced later.
Complete script
Show the complete runnable script
"""Read the supermarket invoices and build a lazy revenue pipeline."""
import flowfile as ff
SALES = "https://raw.githubusercontent.com/edwardvaneechoud/flowfile/main/data/templates/supermarket_sales.csv"
sales = ff.read_csv(SALES, description="Supermarket invoices")
orders = sales.unique().with_columns((ff.col("unit_price") * ff.col("quantity")).alias("revenue"))
preview = orders.select("invoice_id", "city", "revenue").sort("invoice_id")
preview.flow_graph.print_tree()
print(f"{sales.collect().height} rows read")
print(preview.collect())
# Alternative setup: add the reader to a graph you already have.
graph = ff.create_flow_graph()
sales_on_graph = ff.read_csv(SALES, flow_graph=graph)
Recap
You have invoice rows, a revenue column, and a graph describing how they were produced. Next, make the cleaning rules editable and see when an expression becomes a code node.
Next: Clean and shape