Keynote: Working Efficiently with Big Data in Text Formats
summary
Continuum Analytics open sources the CSV/JSON library TextAdapter from its commercial IOPro suite, adds row-position indexing stored locally, and demonstrates concrete speedups: an S3 CSV that drops from ~60s to ~15s (4×) and a gzipped 13M-line file (110 MB compressed, ~2 GB raw) that reads ~20× faster for late-row seeks. Type inference is brittle but converters and regex parsing produce typed numpy arrays or pandas frames, and plans include Dask integration and further indexing features.
AI-written summary · about
chapters (9)
- 0:00Big data text formats
- 1:26Text Adapter library
- 6:51Indexing speeds access
- 13:54Data type inference
- 21:23Converter functions
- 24:16Regex parsing
- 29:16Text Adapter future
- 33:08Dask and Numba integration
- 35:55Open sourcing
related talks
20mGive me the Data
40mFast Access To Your Complex Data #bbuzz
01h 08mUsing and contributing to the data.table package for efficient big data analysis
50mFast Analytics On Big Data
47mFun with BIG Data with various databases
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
05mLightning Talk: A Fast, Concurrent Data Loader for Time-Series Data