Polars 2.0 makes SQL a first-class surface, with disk-spilling for jobs bigger than RAM

Polars, a Rust-built open-source library for working with tabular data, shipped version 2.0 this week with two structural changes: SQL is now a first-class way to use the engine, and the runtime can spill data to disk so a single job no longer has to fit in memory.

Both moves target the sprawl that usually pushes data teams toward a paid warehouse: one Python dataframe library, one separate SQL engine, and a cluster when data outgrows RAM. Polars is collapsing that stack into a single free tool.

The release also adds a Map dtype, stricter type checks, and broader SQL coverage with join reordering, common-subplan elimination, and dynamic predicates via bloom filters.

The post ships TPC-H and TPC-DS comparison runs against DuckDB 1.5.6, DuckDB 2.0 alpha, and DataFusion 54.0.0, run on a disclosed rig (c7a.4xlarge and c7a.metal instances, 5 hot runs per query, separate processes, 60-second timeout, file cache cleared between engines). The numbers are Polars's own, run on its own benchmark rig against specific builds, and DataFusion did not finish every query in the Polars-run comparison.

Out-of-core support is described as "initial" in the release post; scope, stability, and spill-cost overhead are still to be verified. Hacker News discussion treats Polars as the de facto Pandas replacement for analytic workloads, which validates adoption rather than serving as benchmark evidence.

The shift underneath the version bump: one open-source engine now covers dataframe workflows, SQL analytics, and jobs larger than RAM.