Need for speed: the data manipulation version
summary
Work with multi‑gigabyte CSVs (600MB, 2.5GB, 5GB, ~7GB) and avoid the repeated “cannot allocate a vector” crashes by changing approach: benchmark on the actual dataset (she runs 50×), prefer data.table for import and export and for fast filtering/counting, use tidyverse/dplyr for readable complex mutates and case_when, and expect some operations (unique counts, simple adds) to show little difference. Big wins come from strategy: drop unused columns, subset before joining, and replace massive weekly full joins with targeted filters.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 4:56Speed beyond CPU processing
- 8:06Brain processing time importance
- 16:52CPU processing challenges
- 21:25Benchmarking data.table vs dplyr
- 29:13Combining packages effectively
- 37:10Code strategy over functions
- 41:22Advanced speed optimization
- 43:07Summary and Q&A
