Build Your Own DataFrame

A hands-on course that takes apart a real dataframe engine — pyfloe — and teaches you the architecture behind Polars, Spark, and every serious query engine.

This isn't a course you read — it's a course you build with. Every code example is editable and runnable. Change things. Break things. That's the point.

Start the course

Course outline

0

Introduction: Meet pyfloe

A zero-dependency, lazy dataframe engine in pure Python — and the teaching vehicle for this entire course.
1

The Context & Foundation

Why DataFrames exist. Eager vs. lazy execution. Generators and the volcano model. Why lambdas are a dead end for optimization.
2

Python Magic — Dunder Methods & the Expression AST

Hijacking Python's operators so pf.col("x") + pf.col("y") builds an inspectable tree instead of running a computation.
3

The Engine Room — Plan Nodes & Batched Execution

Plugging expressions into plan nodes. Batched execution for performance. The LazyFrame wrapper. Schema propagation without data.
4

Advanced Algorithms — Hashing, Grouping & Joins

Hash joins, join variants, hash aggregation with running accumulators, and sorted-stream alternatives with constant memory.
5

The Query Optimizer (Capstone)

Filter pushdown and column pruning — the optimizer walks the plan tree, inspects expression trees, and rewrites the plan.
+

Interlude: Wiring It Together

Wire everything into a complete mini-pipeline and watch it run end to end.
+

Interlude: Why I Think This Is Interesting

A personal reflection on why understanding the internals matters.
+

Deep Dive: Streaming I/O

Factory-based generators, constant-memory file reading, and type inference from streaming data.
+

Epilogue: Where Do You Go From Here?

Next steps, further reading, and ideas for building your own engine.

Core path: Introduction → Modules 1–5 (with the Interlude between 4 and 5). This is the complete arc.
Optional deep dives: The “Why” interlude can be read at any point. The Streaming deep dive assumes Module 3. The Epilogue assumes Module 5.

This course is built around pyfloe — a lazy dataframe library written entirely in pure Python. No C extensions, no Rust bindings, no dependencies. The same architectural patterns as the big libraries, in a codebase small enough to read in an afternoon.