Cross-Server Data Joins on Slow Networks with Python

on YouTube ↗29m

" summary

Covers techniques and Python tools for querying and joining datasets that live on different machines over slow networks, including approaches for performing cross‑server joins when datasets are too large to fit in local memory. Names no specific libraries in the abstract but promises to discuss several packages and practical strategies for minimizing data movement between a laptop and remote servers.

AI-written summary · about

" chapters (10)

1. 0:00Start
2. 6:41SCENARIO
3. 8:01SOLUTION: GREAT
4. 11:53SUMMARY: GREAT SOLUTION
5. 12:44SOLUTION: NOT BAD
6. 15:47SUMMARY: NOT BAD SOLUTION
7. 16:59SOLUTION: UGLY
8. 17:27HASH JOIN THEORY
9. 22:13DEMO
10. 26:33SUMMARY: UGLY SOLUTION

" related talks

• 37mAccelerating Python Analytics by In-Database ProcessingEdouard Fouché · PyData Berlin 2016
• 49mHow to Combine Data from Different DBMSs and Do It EfficientlyVitaliy Isaev · HighLoad++ 2024
• 34mJoining Heterogeneous Databases is a reality, not a MythIbrar Ahmed · Percona Tech Days 2020
• 37mThe Python Ecosystem for Data Science: A Guided TourChristian Staudt · PyCon.DE 2017
• 47mJOINs Are Slowing You Down: Why Spark and Trino Won’t Replace Your DWHAleksey Dmitriev · Saint HighLoad++ 2025
• 26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and DatafusionMimoune Djouallah · PyCon AU 2025
• 30mParallel computing in Python: Current state and recent advancesPierre Glaser · EuroPython 2019
• 28mPython: Ludicrous mode (with Django)Tyson Clugg · PyCon AU 2017