DataComPy - Dataframe Comparisons made Explicit
summary
DataComPy claims 1.1 million monthly installations and 27 million downloads so far in 2025 and explicitly compares two tables or dataframes by outer‑joining on keys, running NumPy isclose on intersecting columns, and emitting a report with match percentages, sample mismatches and maximal differences. It supports Pandas, Polars, Spark, Snowpark, Dask, Ray, Snowflake and DuckDB; implements native Snowflake comparisons via Snowpark to keep compares consistent and to scale past pandas limits (pandas fails ~50M rows, polars ~500M with 16 CPUs/64GB).
AI-written summary · about
chapters (6)
related talks
39mMastering DataFrame Diffing Techniques
01h 55mAll Them Data Engines: Data Munging with Python circa 2023
18mLooking under the hood - demystifying data tools
30m"Better dataframes"
35mHow many dataframe libraries do you need to change a lightbulb?
35mA Tour of the Many DataFrame Frameworks
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
30mPandas 2, Dask or Polars? Tackling larger data on a single machine