Small Big Data: using NumPy and Pandas

on YouTube ↗01m

" summary

Shows how to process datasets larger than RAM on a single machine using concrete NumPy and Pandas techniques, not cluster handwaving. Explains that running out of memory can make jobs 50–100x slower or crash, and demonstrates practical fixes: numpy.memmap and dtype packing, pandas.read_csv(chunksize) with streaming aggregations, and on-disk columnar formats. Presents working recipes that let tens-to-hundreds-of-gigabyte workloads complete without a cluster.

AI-written summary · about

" related talks

• 34mSmall Big Data: using NumPy and Pandas when your data...Itamar Turner Trauring · PyData New York City 2019
• 26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...Itamar Turner Trauring · PyCon US 2020
• 35mWorking with data in Python for the Aspiring Data ScientistsJean-Pierre Voogt · PyConZA 2019
• 14mKeynote: Scalable Python We Can AffordWes McKinney · Ray Summit 2020
• 01h 17mPandas from the Inside / "Big Pandas"Stephen Simmons · PyData Berlin 2017
• 35mIdiomatic PandasMatt Harrison · Global Artificial Intelligence Virtual Conference 2022
• 01h 02mPandas from the insideStephen Simmons · PyData London 2016
• 01h 22mUsing Dask for Parallel Computing in PythonSkipper Seabold · PyData Chicago 2016