Scale By the Bay 2021
2021-10-28..2021-10-29
-- online --
[Scale By the Bay 2017, Scale By The Bay 2018, Scale By The Bay 2019, Scale By the Bay 2020, Scale By the Bay 2021, Scale By the Bay 2023]
| title | author | length |
|---|---|---|
| Multi Dimensional Clustering with Z Ordering | Russell Alexander Spitzer | 25m |
Shows that multi-dimensional file skipping is practical by computing a per-row Z-Ordering value (by interleaving bits of lexicographically encoded columns) and… | ||
| Distributed Systems with Swift | Konrad 'ktoso' Malawski | 30m |
Language-level Swift 5.5 actors extend to an experimental distributed actors feature plus an open‑source cluster runtime, packaged as swift-distributed-actors… | ||
| Next Generation of Scala | Adam Fraser | 30m |
Argument that Scala's next generation is libraries that are deeply principled and deeply pragmatic and that ZIO is a concrete example: ZIO exposes a ZIO[R,E,A]… | ||
| What the heck is gevent? | Shiv Toolsidass, Roy Williams | 30m |
Gevent is a coroutine-based concurrency layer that uses greenlets and runtime monkey patching to make blocking libraries yield, letting Python microservices… | ||
| Solving the Rubik's Cube with Group Theory | Stewart Stewart | 31m |
Scala code uses the Group typeclass in Cats plus a small custom permutation DSL to model the cube as a subdirect product of an 8-corner permutation and a… | ||
| Making the Transition to Scala 3 | Seth Tisue | 29m |
Scala 3 is final as of May 2021 with 3.1 just out and it reuses the same stdlib jar so most 2.13 code runs unchanged; the compiler accepts nearly all Scala 2… | ||
| The Hardware vs Software | Anna Goldie, Bryan Cantrill, Vitaly Gordon, David Kanter, Alexander O'Connor | 01h 03m |
Debate centers on whether compute becomes a utility or stays decisive: panelists cite compute doubles every 3.4 months and MLPerf as the concrete meter of… | ||
| Semantic Search and Neural Information Retrieval | Amin Ahmad | 30m |
Covers building a production neural search pipeline that replaces brittle term expansion with dense embeddings and nearest-neighbor lookup, demonstrated by… | ||
| Scale By the Bay 2021 by Bill Venners | Bill Venners | 02m |
A new Programming in Scala, Fifth Edition arrives with coverage of Scala 3, printed copies delivered a week ago, and the book documents the concrete language… | ||
| Introducing Hasktorch | AUSTIN HUANG | 27m |
Hasktorch is a Haskell library that binds libtorch (the PyTorch C++ backend) to give GPU support and automatic differentiation while exposing both an untyped… | ||
| Distributed Tracing with Swift | Tim Condon | 30m |
Project implements distributed tracing in server-side Swift using the Swift Distributed Tracing API and exports to back ends like Jaeger/OpenTelemetry and AWS… | ||
| Deploying & serving optimized ML pipelines using graalvm | Adam Gibson | 32m |
Claims that GraalVM plus Eclipse Deeplearning4j and JavaCPP compiles an entire ML pipeline — Python preprocessing, optimized operators, and native math… | ||
| Activity schema data modeling using a single table | Cedric Dussud | 27m |
Claims that business analytics fit into a single time‑series table with 11 columns (no JSON) where every source row becomes a customer activity (pageview… | ||
| Functional Programming + Dependent Types ≡ Verified Linear.. | Ryan Orendorff | 34m |
Claims that representing matrices as functions plus dependent types yields verified linear algebra with proofs in Agda; implements matrices as linear functions… | ||
| Pure Scala Approach for Building Frontend and Backend Apps | Taro Saito | 29m |
Claims a pure-Scala stack that uses Airframe RPC to expose Scala traits and case classes as RPC endpoints, serializes via airframe-codec (MessagePack/JSON)… | ||
| Advanced Features of Scala 3 | Bill Venners | 32m |
Covers concrete Scala 3 features and how to use them: opaque types (alias that avoids boxing at call sites, with example logarithm and posint hierarchies)… | ||
| Live Coding: Building Stateful and Scalable Reactive Streams... | Nolan Grace | 30m |
Live-codes a working pipeline that reads partitioned Kafka quotes, groups by symbol, and computes 10‑second windows (max 10 substreams) to emit max/min and… | ||
| Proxies, Gateways, and Meshes: Cloud Connectivity 101 for... | Viktor Gamov | 34m |
Compares concrete uses: API gateways (example: Kong with plugins) handle auth, TLS termination, rate limiting, request transformation, mocking, documentation… | ||
| How We Built SQL Rollups on Streaming Data | Tudor Bosman, Karen Li | 22m |
Rockset implements SQL rollups on Kafka and Kinesis by running a decomposed SQL rollup query at ingest and finalizing aggregates at read time, cutting storage… | ||
| What I Learned about FP when Writing a Book About It | Sergei Winitzki | 31m |
Derives functional-programming results from first principles and shows which theories actually matter for writing Scala code: proves which type signatures are… | ||
| Scale By the Bay 2021 intro video by Sergei Winitzki | Sergei Winitzki | 03m |
A nearly finished open source book titled The Science of Functional Programming provides full source code on GitHub, program examples, diagrams and scripts to… | ||
| Lessons Learned from 15 Years of Scala in the Wild | Dean Wampler | 30m |
Scala 3 claims to fix pain points from ~15 years of real use by making implicits explicit with using/given context and true extension methods, adding opaque… | ||
| Algebraic Data Types, Categorically | Vlad Patryshev | 31m |
Claims that algebraic data types are precisely models of Lawvere (algebraic) theories and that these models appear as polynomial functors with arrows p(X) → X… | ||
| Rethinking scalable machine learning with Spark ecosystem | Adi Polak | 23m |
Argues that an end-to-end ML pipeline runs inside the Apache Spark ecosystem by doing ETL, feature engineering and model training at scale, then bridging into… | ||
| Scale By the Bay 2021 intro video by Alyona Medelyan | Alyona Medelyan | 01m |
Company builds a SaaS product that analyzes large volumes of unstructured feedback using NLP, moves from bootstrapped prototype to YCombinator and VC funding… | ||
| Developing Declarative Backends | Ramnivas Laddad | 30m |
A declarative backend language compiles to a runnable backend that exposes GraphQL API with models, JWT role rules, and external-service bindings written in… | ||
| Startups and the ML landscape | Stephen Merity, James Cham, Lisha Li, Alexy Khrabrov | 01h 09m |
Panel argues practical paths: prioritize ship a de-risking prototype (example: a faces makeup mobile app before full storytelling), test on consumer channels… | ||
| Scaling your Kafka streaming pipeline.... | Opher Dubrovsky, Ido Nadler | 32m |
Nielsen Marketing Cloud runs a multi-petabyte Kafka pipeline that ingests ~25 billion events/day (≈300k/sec, peaks to 5M/sec, ~200 TB/day) across 60 brokers… | ||
| Scale By the Bay 2021 intro video by Paweł Skrzypek & Anna Warno | Pawel Skrzypek, Anna Warno | 01m |
Overview of concrete deliverables and methods for using state of the art machine learning models for CDS forecasting and folding those forecasts into portfolio… | ||
| Building an ML Platform from Scratch | Alon Gubkin | 29m |
A minimal ML platform is built end to end: a cookiecutter model template generates a FastAPI model server and training scaffolding, experiments and pickled… | ||
| Scale By the Bay 2021 intro video by Jennifer Reif | Jennifer Reif | 01m |
A short pitch for a session that shows Neo4j plus Java applications building a working app that queries and mutates connected data, with a live demo that… | ||
| Single Serving Values with Linear Types in Scala | James Earl Douglas | 32m |
Linear types require values be used exactly once and the implementation here adds that check to Scala via a Scalafix lint called linear scala that marks types… | ||
| Scale By the Bay 2021 intro video by Ido Nadler | Ido Nadler | 01m |
Conference intro states a production data stack that runs 250 Billion events/day on AWS and describes the actual pipelines and infra built to do it. It lists… | ||
| Scale By the Bay 2021 intro video by Eynav Mass | EYNAV MASS | 01m |
Claims that auto-scaling is not enough and that incremental growth plus sudden peaks created unexpected failure modes at Oribi, which handles billions of… | ||
| Apache Spark Performance Tuning Session with Delight | Jean-Yves Stephan | 29m |
Delight is an open-source Spark monitoring dashboard that injects a driver agent jar, streams encrypted metrics to a hosted backend, and aligns memory, CPU and… | ||
| Keynote, Why and how to care about ethics in ML | Clément Delangue | 28m |
Hugging Face claims over 30,000 models on its platform used by more than ten thousand companies and shows concrete bias in BERT’s fill‑in‑the‑blank results… | ||
| Battle-tested event-driven patterns for your microservices | Natan Silnitsky | 41m |
Wix runs ~2,200 microservices and 2.5 billion Kafka messages per day and uses four battle-tested patterns: Consume and Project (reverse-lookup writer + reader… | ||
| Scale By the Bay 2021 intro video by Opher Dubrovsky | Opher Dubrovsky | 01m |
Concrete cases of Kafka consumer failures under rapid traffic shifts and during broker recovery, with measured numbers from a pipeline handling 250 Billion… | ||
| Scale By the Bay 2021 intro video by Adarsa Sivaprasad | Adarsa Sivaprasad | 02m |
Concrete notes on turning ML prototypes into production: embedding recommendation engines and knowledge graphs into products, building continuous deployment… | ||
| Scaling the Disney Streaming Offer Management Platform... | Anshuman Nayak | 27m |
Disney Streaming grows from 10 million signups at launch to about 116 million subscribers, forcing rework of offer and catalog services to handle multi‑product… | ||
| Making Kubernetes How We Build Things | Rob Richardson | 22m |
Demo shows how to go from local Docker to a running Kubernetes loop with concrete tools and commands: use Docker Compose bridge networking to link multiple… | ||
| Alice and Mad Hatter.... | Roksolana Diachuk, Marianna Diachuk | 30m |
Built a Scala Spark pipeline that turns raw records into cached RDD vectors, applies RDD vector normalization (mean/stddev), trains KMeans (rerunning to avoid… | ||
| Top new CNCF projects to look out for | Annie Talvasto | 26m |
A rapid tour of CNCF projects that names concrete capabilities and live results: Linkerd is a graduated service mesh with an ultralight Rust proxy that… | ||
| The Incremental ETL Architecture | John O'Dwyer | 30m |
Incremental ETL at big scale uses Apache Spark Structured Streaming with Delta Lake tables and Databricks Autoloader to move only new or changed rows, keeping… | ||
| Scale By the Bay 2021 Intro Video by Anatolii Kmetiuk | Anatolii Kmetiuk | 02m |
Anatolii Kmetiuk introduces the Scale By the Bay 2021 conference, giving logistical details (online format, dates Oct 28–29 2021, registration link and contact… | ||
| Tuning Hyperparameters with DVC Experiments | Milecia McGregor | 31m |
Grid search with three hyperparameters and ten values each explodes to 1,000 runs while random search often finds better settings in the same time by sampling… | ||
| Scale By the Bay 2021 intro video by Annie Talvasto | Annie Talvasto | 02m |
Promises a short list of top new CNCF projects that claim to supercharge Kubernetes clusters, and a practical angle on automating Kubernetes cost optimization… | ||
| Scale By the Bay 2021 intro video by Milecia McGregor | Milecia McGregor | 01m |
Demonstrates tuning hyperparameters for any ML project using dvc experiments from iterative.ai, with a concrete workflow that automates grid search and random… | ||
| Reproducible Machine Learning at Scale | Lukas Biewald | 30m |
Shows how teams use Weights & Biases to track datasets, version training recipes, and store run metadata so experiments are auditable and shareable… | ||
| Scale By the Bay 2021 intro video by Russell Spitzer | Russell Alexander Spitzer | 01m |
Explains an implementation of z-ordering inside Apache Iceberg that clusters files with a Z-order curve so multi-dimensional filters avoid full table scans… | ||
| Advanced machine learning time series.... | Pawel Skrzypek, Anna Warno | 31m |
Survey of modern time-series methods with concrete claims: Temporal Fusion Transformer and hybrid architectures (ES-Hybrid, N-BEATS, Tsetlin) repeatedly beat… | ||
| The intersection of Quantum Computing and Distributed Systems... | Karl Wehden | 32m |
Argues that NISQ machines need runtime-based quantum execution pushed nearer hardware (co‑located container runtimes) instead of remote gate loops, reducing… | ||
| Relational Databases Don't call it a comeback! | Rob Hedgpeth | 28m |
Argues that relational databases modernize rather than lose relevance by adding columnar storage for analytics (example: averaging one billion rows reads 200… | ||
| Live coding session with James Douglas | James Earl Douglas | 30m |
Builds a toy effect system in Scala that represents an I/O program as a pure-value program: three effects (env lookup, read line, write) are encoded as traits… | ||
| Scale By the Bay 2021 intro video by Kévin Rauscher | Kévin Rauscher | 01m |
Conference opener announces 28th-29th October 2021 and confirms an online format, gives the registration URL and a partner contact email, and frames the event… | ||
| Continous integration of ML products and UX design | Adarsa Sivaprasad | 27m |
Argues that product design must embed ML lifecycle work not just dashboards, giving concrete examples: a telemetry summarizer for core data aggregation and a… | ||
| Funnel Rocket: A Serverless Query Engine | Elad Rosenheim | 34m |
An open-source engine that runs serverless big-data funnel and sequence queries by composing AWS Lambda, Pandas + Apache Arrow and Parquet on cloud storage… | ||
| Scale By the Bay 2021 intro video by Mary Grygleski and Fabio Tiriticco | Mary Grygleski, Fabio Tiriticco | 02m |
Retrofits a Java app into a reactive flow pipeline that uses back pressure to protect legacy services from traffic spikes, swaps HTTP for RSocket as a… | ||
| Scale By the Bay 2021 intro video by Ryan Orendorff | Ryan Orendorff | 01m |
Claims that functional programming plus dependent types yields verified linear algebra by representing matrices as matrix-free functions between vector spaces… | ||
| Scale By the Bay 2021 intro video by Joshua Arvin Lat | Joshua Arvin Lat | 01m |
Short intro that promises concrete operational patterns for scaling ML: practical recipes from the Amazon SageMaker Cookbook, hands-on use of Amazon SageMaker… | ||
| intro video | Roksolana Diachuk | 01m |
Announcement for a co-presentation that demonstrates how big data engineers and data scientists divide work to build a working machine learning pipeline, from… | ||
| intro video | Lukas Biewald | 01m |
An experiment-tracking platform called Weights & Biases is now used by thousands of ML teams deploying models across automotive, pharma, ag‑tech and B2B… | ||
| Introducing Spark Cyclone Accelerating Spark with... | Eduardo González | 30m |
A Spark Cyclone plugin offloads Spark SQL to the NEC SX‑Aurora TSUBASA vector engine, claiming up to 48GB HBM2, 1.5 TB/s per card and ~6 TFLOPS (49 TFLOPS with… | ||
| Retrofit your Java App with a Reactive Flow | Mary Grygleski, Fabio Tiriticco | 27m |
Demo retrofits a monolithic JVM workflow into a reactive pipeline using Kafka as source/sink, Akka Streams on the producer side, an RSocket link, and a Java… | ||
| Pouring Coffee into the Matrix - Building Java Apps on Neo4j | Jennifer Reif | 31m |
Builds a small Java service that stores Java versions and their diffs in Neo4j, using Spring Data Neo4j and the official Neo4j Java driver over Bolt to a Neo4j… | ||
| Convergence of AI, Simulations and HPC | Anima Anandkumar | 25m |
Introduces a Fourier Neural Operator that learns function-to-function maps using global convolution in the Fourier domain, trains on low-resolution (e.g… | ||
| intro video | John O'Dwyer | 01m |
Explains the incremental ETL claim as a pattern that processes only changed data so pipelines run faster and cheaper, and shows it implemented on Databricks… | ||
| Casting the spell: Druid in practice | Itai Yaffe, Yakir Buskilla | 28m |
Uses Apache Druid with DataSketches to serve real-time dashboards on tens of terabytes ingested daily, relying on Theta sketch (KMV) for approximate distinct… | ||
| intro video | Itai Yaffe, Yakir Buskilla | 02m |
ai.bythebay.io Nov 2025, Oakland, full-stack AI conference We are happy to invite you to the 9th conference Scale By the Bay!
Format: Online
Dates… | ||
| intro video | Marianna Diachuk | 01m |
Conference intro previews a case study that builds a prediction pipeline around a narrative called Alice and the Mad Hatter, contrasts a data scientist… | ||
| A Journey of Migrating Data Platform from Data Center to Cloud | Lei Gao | 31m |
Migration moves a 19+ PB data lake with 19+ PB and 2 trillion messages/month from colo to public cloud, replacing Mesos-based runtime with cloud-native stacks… | ||
| Keynote: How to start a startup for AI Engineers | Alyona Medelyan | 29m |
Thematic builds an AI pipeline that starts with unsupervised theme discovery over surveys, tickets and reviews, adds a visualization layer like Tableau for… | ||
| intro video | Taro L Saito | 01m |
Demonstrates Airframe RPC as a concrete bridge between JVM services and browser code, using Scala.js to compile the client and the Airframe framework as a… | ||
| Functional streams for factorians | Kévin Rauscher | 27m |
Implements a mini Factorio factory using FS2 (Functional Streams for Scala) and a bounded Queue via Cats Effect, composing drill -> belt -> furnace ->… | ||
| Troubleshooting Techniques for Complex Codebases | Anatolii Kmetiuk | 30m |
Fixing Scala 3 (Dotty) compiler bugs focuses on getting fast, repeatable experiments: the workflow standardizes every report into an issue reproduction script… | ||
| Building Complete State of the art Natural Language Processing | David Talby | 30m |
Three production tools built on Spark NLP enable fast end-to-end projects: a Python NLU library that gives one-line load and predict access to over 4,000… | ||
| Prepare Your System To Scale OR Why Auto Scaling Is Not Enough | EYNAV MASS | 28m |
Oribi handles 100M→300M events/day and survives a sudden 10x spike in one hour by combining Kafka, Elasticsearch, Apache Druid and Kubernetes with specific… | ||
| 10 Things You Should Know About Quantum Computing and ML | Chris Fregly | 29m |
Explains practical quantum ML state: NISQ devices run hybrid circuits using Hadamard and CNOT gates and libraries like Qiskit, TensorFlow Quantum, PennyLane… | ||
| Transformers End to End Experience training... | Alexander O'Connor | 34m |
Autodesk describes practical results using Hugging Face models for support routing and e‑commerce tasks, favoring zero-shot classification and weak supervision… | ||
| intro video | Elad Rosenheim | 02m |
Funnel Rocket is an open source platform that replaces always-on, expensive clusters with a cloud-native serverless architecture, claiming significantly lower… | ||
| Location Based Data Engineering for Good with PySpark:Graphs | Evan Chan | 28m |
Pipeline shows how PySpark plus Kedro reads cloud-stored pings, sessionizes by device, coalesces adjacent points with window functions, and infers trips using… | ||
| Responsible AI Challenges and Recommendations | Ricardo Baeza-Yates | 29m |
Survey of five concrete problems: COMPAS racial bias and bail prediction models that claim a 25% crime reduction / 42% prison reduction but amplify minority… | ||
| Lat Designing and Building Complex Machine Learning.... | Joshua Arvin Lat | 34m |
Shows a concrete stack: SageMaker Debugger rules trigger SNS alerts when training loss stalls, automated hyperparameter tuning runs many parallel jobs (and can… | ||
| nothing matches | ||