Data @Scale 2024
who it's for
data engineers, data platform engineers, machine learning engineers, ml platform engineers, site reliability engineers, engineering managers
topics
Nimble file format; Velox vectorized engine; Velox C++ execution library; real-time LLM incident assistant (Llama 2 7B); unified Data Graph and discovery API; Amazon Q to AWS Glue integration; Apache Beam ML transforms and GPU pooling; DuckDB in WASM for interactive analytics; Snowflake LLM data pipeline deduplication; Aurora multi-AZ with Go microservices
| title | author | length |
|---|---|---|
| Opening Remarks | Maor Kleider, Jelena Pjesivac-Grbovic | 02m |
Conference frames practical engineering challenges across vendors including AWS, Google, Razorpay, Mduck, Snowflake and Meta while focusing on concrete systems… | ||
| The AI-First Data Infrastructure | Barak Yagour | 18m |
Meta develops a new file format called Nimble for encoding and workload optimization and a vectorized engine Velox to process massive datasets; both are open… | ||
| Navigating Data's Next Great Shift | DJ Patil, Daniel Francisco | 27m |
Core claim: messy pipelines remain the big bottleneck — clean data as foundation — while teams that invest in hygiene unlock GenAI and vision models faster… | ||
| Scaling Meta’s Infra with GenAI: Journey to faster and smarter Incident Response | Diana Hsu, Mohamed Farrag | 20m |
Meta builds a real-time incident onboarding and root-cause pipeline using a fine-tuned Llama 2 7B model plus an internal LM assistant that reads chat, logs and… | ||
| Composable Data Management Systems | Pedro Pedreira, Amit Purohit | 17m |
Meta builds a reusable execution layer called Velox C++ execution library (open sourced in 2022) that executes fully optimized plans and integrates into… | ||
| Demystifying the Data Stack of the Largest and Fastest Growing Payment Gateway in India | Murali Brahmadesam | 19m |
Razorpay handles ~$150B TPV across 5M+ businesses while maintaining 99.99% uptime by using Aurora multi‑AZ with proxy servers, migrating a PHP monolith to Go… | ||
| Large-Scale Data Graph: Scale and Optimize Privacy & Security in Offline Data Systems | Can Lin, David Taieb | 18m |
Meta builds a unified Data Graph that maps warehouses, stream processors and observability as vertices and flows, storing class/type, global and internal IDs… | ||
| Case Study in Bridging Production Software and Data Practices for LLM Model Training | Nathan Wiegand, Kelvin So | 22m |
Snowflake hosts the entire LLM data pipeline: data lands as tens of terabytes from Common Crawl and Hugging Face, is split into a file metadata/file contents… | ||
| Smarter, faster Data analytics with generative AI and Machine Learning | Santosh Chandrachood | 21m |
Shows a concrete pipeline: using Amazon Q for data integration to turn English into AWS Glue jobs that join S3/RDS/Redshift sources and generate optimized… | ||
| Beam Up Your GenAI Usage: Usability, Efficiency, Reliability | Ahmet Altay | 19m |
Apache Beam ML claims concrete results: Spotify cuts a 110‑minute podcast preview build to under four minutes and reports 30x speedups, Credit Karma runs… | ||
| Taking Flight with Interactive Analytics | Frances Perry | 16m |
Argues that most analytics use tiny working sets—99% of queries under 10 GB, 90% under 100 MB—and shows a browser demo that runs DuckDB in WASM for local… | ||
| Fireside Chat: Evolution of AI-first Data Infrastructure | Manju Anand | 22m |
Discussion drills into concrete changes: a move from schematized tables to media and multimodal stores, video decoding optimizations, and a new ML-optimized… | ||
| Live Q&A Session #1 | Manju Anand | 26m |
Composable data tooling is an extensible engine-agnostic library that runs inside different compute engines rather than replacing them, with migrations varying… | ||
| Data @Scale 2024 | Amazon Web Services | 05h 34m |
Meta reports a 3x speedup moving Presto workloads to a C++ execution library called Vu/Velox, introduces a Nimble file format and an MTA V1 accelerator while… | ||
| Closing Remarks | Maor Kleider, Jelena Pjesivac-Grbovic | 01m |
Quick recap of concrete takeaways: current work focuses on solving scale, diversity, quality for data used in generative AI, and on building pipelines that… | ||
| Live Q&A Session #2 | Manju Anand | 21m |
A live Q&A covers concrete wins and limits: a corporate data graph faces idiosyncratic systems, needs precise accuracy and coverage metrics, and solves… | ||
| nothing matches | ||