Python and pandas as back end to real-time data driven applications

on YouTube ↗40m

" summary

Built a web app using Python and pandas with a Flask + AngularJS front end, Hadoop for raw storage and postgress for serving. Logs shrink from 40 terabytes to 0.5 terabyte by hourly aggregation; a first iteration loads 3 GB into memory. Geospatial filtering uses vectorized NumPy operations and temporary tables to avoid giant IN lists. Query latency drops from ~20 minutes to 5–6 seconds (often milliseconds) and the system works for low-concurrency real‑time use.

AI-written summary · about

" chapters (14)

1. 0:00Real-time data applications
2. 3:25Client project overview
3. 7:23Big Data challenges
4. 10:17Real-time retrieval with pandas
5. 11:45Architecture: Flask and AngularJS
6. 14:35Pandas operations
7. 18:33Prototype with pandas
8. 20:00Advantages and limitations of pandas approach
9. 22:30Switching to PostgreSQL
10. 23:12Geographical data handling
11. 27:37Alternative databases
12. 29:26PostgreSQL optimizations
13. 31:56Data compression
14. 33:01Q&A

" related talks

• 34mSmall Big Data: using NumPy and Pandas when your data...Itamar Turner Trauring · PyData New York City 2019
• 44mUsing Python with Hadoop to create production ready big data applicationsUlrich Zink · PyData London 2016
• 34mPython as a Framework for Analytics and Growth HackingIgnacio Elola · PyData Berlin 2015
• 40mScaling Python - Bank EditionAndrew Fulton, Adam Lewis · PyData NYC 2022
• 14mKeynote: Scalable Python We Can AffordWes McKinney · Ray Summit 2020
• 47mBuilding Scalable Web AppsDavid Cramer · EuroPython 2011
• 01h 16mIntro to Data Analysis with Python Data-tableAna Castro Salazar, Pasha Stetsenko · PyData Los Angeles 2019
• 40mBeyond Pandas: The Great Python Dataframe ShowdownJuan Luis Cano Rodríguez · PyData London 2022