Leveraging a custom CPython data model for high performance...
summary
Build claim: a custom read-only binary Design Data (DD) graph stores topology and derived metrics so engineers load and query entire designs. Implementation detail: a C++ core with a CPython extension module lets Python-driven analysis use Flask, websockets, pandas and visualization. Performance numbers: a complete 10,000-node 50M-edge graph takes 6 minutes and >8 GB in pure Python versus 4 seconds and ~1 GB in C++; typical analysis modules use 16 cores, >0.5 TB RAM, and up to 8 hours, and a DD file loads into Python in under 5 minutes.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:46How we build a microprocessor
- 1:48Microprocessor complexity
- 2:48Micro architecture
- 3:37Optimization
- 4:37Complex design
- 5:19Managing multiple versions
- 6:47Design Data
- 7:27Data Model Integration
- 9:45Python Environment
- 11:18Engineering View
- 12:13Engineering View Example
- 14:58Custom Analysis Example
- 15:50Load Multiple Data Files
- 16:14Developers Experience
- 16:41Debugging
- 17:37Checking for changes
- 18:23Vectorized approach
- 20:24democratized analysis
- 20:43references
- 21:47conclusion
related talks
40mScaling Python - Bank Edition
46mGraph Databases: Talking about your Data Relationships with Python
39mAccelerated Data Science Libraries
37mAccelerating Python Analytics by In-Database Processing
26mGraph Databases will change your (freakin') life
25mProcessing Biggish Data with DuckDB and Python
01h 16mIntro to Data Analysis with Python Data-table
42mOptimizing Code Performance for Python Internals