The Data Analytics Platform or How to Make Data Science in a Box Possible
summary
ING builds a Kubernetes-based data platform aimed to reach 50% of 50,000 employees in two years, hosting multi-petabyte workloads with Spark on Kubernetes, per-user JupyterLab “data science in a box” images built via GitLab/Callisto and pulled in ~20 seconds, a web VDI using XRDP+Guacamole completed in a week, Ceph storage fronted by a custom Roku proxy for short-lived tokens and Ranger auth, Amundsen for searchable metadata lineage, and Presto queries hitting ~10 billion records/sec without Roku bottlenecks.
AI-written summary · about
chapters (13)
- 0:00Intro
- 5:00Architecture
- 7:55DOP Portal
- 12:03Compute
- 14:00Data Science in a Box
- 15:40DAP Portal
- 19:37Security
- 21:41How does it work
- 23:51Team
- 24:54Questions
- 26:36Docker Image
- 28:11Performance
- 31:11Munson and Atlas
related talks
27mData Analytics Democratization: How ING Data Analytics Platform Bootstraps New...
17mDemocratising data analytics
26mRe-architecting the Data Platform with Spark on Kubernetes
05mHow to Simplify Data Scientist’s Day in Huge Company Using Kubernetes
25mEmpowering AI-Driven Drug Discovery: Overcoming Challenges in Building a ML Platform on Kubernetes
45mBuilding an Analytics Platform
44mEmpowering Data Science with Kubernetes | Samuele Chiocca and Andrea Rota
21mPutting big data self-service analytics in the hands of business people