Accelerating Python Analytics by In-Database Processing
summary
Claims that pushing Python operations into SQL speeds analytics by keeping data in-place and using DB features. The demo builds an ibmdbpy prototype that translates Pandas/Scikit-learn-like calls into SQL, runs in-database algorithms including Kmeans and Linear Regression, and computes correlation/covariance without downloading data. Benchmarks show the crossover at about half a million rows where in-database wins and a local run fails at ~18 million rows, and the prototype executes the pushed queries and returns results as data frames.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:04Talk outline and motivation
- 3:52Python analytics limitations
- 4:54In-database computation advantages
- 6:19IBM Db2 and IBM DTP intro
- 7:24Python vs SQL for analytics
- 9:11SQL push-down and IBM DTP design
- 12:46IBM DTP basic demo
- 14:56IBM DTP advanced analytics demo
- 17:46Non-destructive data manipulation in DTP
- 18:08In-database ML with DTP
- 20:19Feature selection in DTP
- 22:07Performance comparison: in-db vs in-memory
- 25:01IBM DTP deployment and availability
- 25:23Future work for IBM DTP
- 26:49IBM DTP benefits summary
- 28:38Q&A
related talks
39mPython and SQL: Better Together, Powered by DuckDB
26mPython + MPP Database = Large Scale AI/ML Projects in Production Faster
41mMassively Parallel Processing with Procedural Python
37mIn database Machine Learning with Python in SQL Server
29mBlazing-Fast Python in Your Database: Unlocking Data Science at Scale with Exasol
30mDuckDB: Bringing analytical SQL directly to your Python shell
40mScaling Python - Bank Edition
41mFaster SQL with pandas and Apache Arrow