Optimizing SQL + Python Pipelines for Data Science
summary
Clear, actionable takeaways: flipping the FROM table can save seconds at scale (one example yields a six‑second gap), moving complex logic into temp tables beats nested subselects for optimization and debuggability, prefer union all when duplicates don’t matter, do lead/lag and heavy joins in SQL rather than pandas, export a trimmed result to a compressed CSV to read into pandas faster, and enabling batch writes (e.g. rewriteBatchedStatements=true) cuts multi‑hour writes down to minutes.
AI-written summary · about
chapters (13)
- 0:00Start
- 0:44Introduction
- 2:49SQL and Python pipelines
- 3:54SQL best practices
- 5:39Temp tables for performance
- 7:23Indexing and query tips
- 9:25Join data before Python
- 11:12Data volume vs ML trade-offs
- 13:15Feature engineering: SQL vs Python
- 15:42Pandas strengths and limits
- 16:43SQL to pandas transfer
- 18:08Batch writing for exports
- 19:10Summary and Q&A
related talks
32mUnlearning SQL
35mData Processing Pipelines for Small Big Data
37mAccelerating Python Analytics by In-Database Processing
47mHow to pick the best Data transformation strategy
16mExploring Query Optimization: How a few lines of code can save hours of time
25mOptimize your Data Pipeline without Rewriting it
28mIdiomatic Pandas
50mServerless SQL Pipelines