Flowfile

An open-source data platform with a visual pipeline builder, AI assistant, data catalog with Delta Lake storage, scheduling, Kafka ingestion, sandboxed Python execution, and a Polars-compatible API — installed with a single pip install.

New here? Start with what Flowfile is →

Flowfile AI Assistant in action

What's inside

A complete data platform built on Polars — every part works with every other part

Visual Pipeline Builder

40+ nodes for joins, filters, aggregations, fuzzy matching, pivots, and more — with a data preview at every step. Missing one? Build your own visually in the Node Designer.

AI Assistant

Build pipelines from natural-language prompts, ask questions about a flow you didn't write, and turn failed runs into one-paragraph diagnoses. Bring your own provider — Anthropic, OpenAI, Google, Groq, OpenRouter, or a local Ollama.

Data Catalog & Delta Lake

Tables stored as Delta Lake with version history and time travel. Namespaces, lineage, run history, and virtual tables in one place.

Analyze & Explore

Query any catalog table in the SQL editor, explore results in Graphic Walker visualizations, and save the charts next to the data.

Scheduling & Triggers

Run flows on intervals or trigger them when catalog tables update. Built into the catalog — not a separate orchestration tool.

Kafka, Databases & Cloud Storage

Ingest from Kafka/Redpanda as a canvas node. Read and write S3, Azure Data Lake, and GCS. Connect PostgreSQL, MySQL, and SQLite.

Sandboxed Python

Run arbitrary Python in isolated Docker containers. Use any library — the output flows back into the pipeline.

Code Generation & Python API

Export visual flows as readable Python/Polars scripts, or build pipelines in code with a Polars-compatible API and open them on the canvas.

One pipeline, two ways

The same sales pipeline — cleaned, filtered, and aggregated — on the canvas and in Python. Both produce the same flow; switch whenever you like.

On the canvas

  1. Read — load supermarket_sales.csv
  2. Drop Duplicates — remove repeated invoice rows
  3. Filter — keep bulk orders: [quantity] > 7
  4. Group By — per city, the sum and median of gross income

In Python

import flowfile as ff

df = ff.read_csv("data/templates/supermarket_sales.csv")

result = (
    df.unique()
    .filter(ff.col("quantity") > 7)
    .group_by("city")
    .agg(
        ff.col("gross_income").sum().alias("total_income"),
        ff.col("gross_income").median().alias("median_income"),
    )
)

print(result.collect())

Also available in the app: Create → From template → "Sales pipeline". The flow and the code above are validated by the test suite on every commit.

Where to start

Pick the guide written for the way you work

Build flows visually

Turn messy exports into pipelines everyone can see and re-run.

Coming from Excel

VLOOKUP, pivot tables, and IF-formulas, translated to flows.

Your data lives elsewhere

Warehouse, S3, Kafka, GA — work with data where it already is.

Analyze your data

From question to chart you trust — and it refreshes itself.

Write Python

A Polars-compatible API that builds visual flows as a side effect.

Run Flowfile for a team

Auth, secrets, sharing, backups — the operator's route.

Free and open source

One install gives you the visual editor, the Python API, the catalog, and all services.

pip install flowfile
Quickstart Guides by audience

MIT licensed, developed on GitHub