Berlin Buzzwords 2021
2021-06-15..2021-06-17
Berlin, Germany
[Berlin Buzzwords 2012, Berlin Buzzwords 2015, Berlin Buzzwords 2016, Berlin Buzzwords 2018, Berlin Buzzwords 2019, Berlin Buzzwords 2020, Berlin Buzzwords 2021, Berlin Buzzwords 2022, Berlin Buzzwords 2023, Berlin Buzzwords 2024, Berlin Buzzwords 2025, Berlin Buzzwords 2026]
| title | author | length |
|---|---|---|
| Democratizing Climate Science | Sherin Thomas, Dharini Chandrasekaran | 44m |
SpaceML builds a production reverse‑image search over NASA GIBS covering 20+ years and petabytes of imagery, using a GIBS downloader to tile multi‑band data… | ||
| Scaling Facets to the Stars 🌟 | Shikhar Srivastava | 31m |
Explains how Apache Solr range facets are engineered to return daily counts over years at sub-second latency by prewarming the filter cache and aligning query… | ||
| Future of Lucene's MMapDirectory: Why use it & what's coming with Java 16 & later? | Uwe Schindler | 35m |
Discusses Lucene's use of MMapDirectory for disk access to avoid user‑space copies and use the OS page cache, and explains current 32‑bit‑style addressing… | ||
| Docker Who. Small Containers Through Time and Space | Dmitry Chuyko | 31m |
Alpine Linux support is now in OpenJDK via the Portola work merged around JDK 16, enabling musl-linked JDK builds (used by Liberica JDK) that shrink base… | ||
| Learning to Judge | Arne Vogt, Andrea Schütt | 35m |
Otto builds ranking gold standards from customer logs by computing Bayesian-smoothed click probabilities per query and correcting for position bias with… | ||
| Engineering Intent and Intent Measurability | Angel Maldonado | 30m |
Claims that intent is fully measurable give way to specific practices: treat intent measurability as approximate, stop flattening decisions to event sequences… | ||
| How Me2B can improve your search and discovery experience | Pieter Vegt | 28m |
Argues for swapping platform-owned profiles for personal pods so retailers connect into a customer-controlled space via a vendor relationship manager model… | ||
| Casting the Spell: Druid in Practice | Itai Yaffe, Yakir Buskilla | 27m |
Apache Druid powers a pipeline that ingests tens of terabytes daily (10 billion events/day, >5 PB lake, 60 TB/day on S3) using Hadoop-based ingestion with… | ||
| Designing Payloads for Event-Driven Systems | Lorna Jane Mitchell | 30m |
Design payloads for Kafka by sending non-essential metadata in headers, using the key to shard/order across partitions, and keeping the root an object — never… | ||
| The Invasion of Transformers - Boosting Search with Latest NLP | Malte Pietsch | 33m |
Claims that transformer-backed search moves from lab to production with measurable gains using dense passage retrieval (DPR) plus Approx. Nearest Neighbour… | ||
| Deep Learning, Neuroscience and the future of AI | Darjan Salaj | 29m |
Argues that spiking neural networks plus neuromorphic chips (Intel Loihi, SpiNNaker, BrainScaleS) already cut energy and latency by up to 100× on tasks like… | ||
| Applied MLOps to Maintain Model Freshness on Kubernetes | Jeff Zemerick, David Smithbauer | 31m |
A working Kubernetes pipeline that ingests Twitter via an Apache Flink consumer, naively counts hashtags in Redis, classifies movie summaries with a zero-shot… | ||
| Documentation: The Missing Pieces | Adrienne Braganza Tacke | 33m |
Most documentation fails because it assumes knowledge and skips exact steps; fix it by spelling out acronyms, replacing foo/bar with descriptive names, and… | ||
| The Debate: Which Search Engine? | Jo Kristian Bergum, Anshum Gupta, Josh Devins, Charlie Hull | 01h 48m |
Three engines compared by concrete claims and limits: Vespa advertises native tensor support, ANN with HNSW and hybrid term+vector ranking, true real‑time… | ||
| Databases in the Microservices World | Rob Richardson | 30m |
Argues for treating each microservice as owner of its micro-database and automating provisioning, authentication and backups so dozens of heterogeneous… | ||
| Facets of Reproducible research on Kubernetes | Sophie Watson, William Benton | 28m |
Kubernetes plus CI/CD and container images make environments reproducible, but real reproducibility requires concrete steps: use binder or declarative images… | ||
| Taking AI/ML to the next level: Text! | Nick Burch | 31m |
Builds a concrete pipeline that parses documents with Tika, converts abstracts into TF‑IDF vectors, and uses scikit-learn for a quick classifier and k‑means… | ||
| Autoscaling Apache Flink Applications | Robert Metzger | 20m |
Three scaling modes are compared: manual savepoint-based rescaling (stop, take savepoint, restore), Flink 1.13 reactive scaling which auto-expands parallelism… | ||
| Tips and Tricks to Scale Elasticsearch for 1M RPMs and Beyond | Dainius Jocas | 41m |
Shows concrete fixes and measurable wins: change integer fields to keyword IDs reduces p99 by ~20 ms on term-heavy filters; avoid date math and rewrite date… | ||
| Search and Sushi; Freshness Counts | Jo Kristian Bergum | 28m |
Open source Vespa implements a mutable in‑memory index backed by an immutable disk index and a transaction log so updates are durable and immediately visible… | ||
| Document classification search; joins vs payloads | Kevin Watters | 29m |
Adds inequality ops to the payload check query parser (see LUCENE-9659 / SOLR-14787) and encodes ML classification confidences as payloads so queries can test… | ||
| It's Time To Stop Using Lambda Architecture | Yaroslav Tkachenko | 28m |
Argues that Lambda’s dual systems are avoidable by using Apache Kafka compaction or tiered storage for long retention, Kafka transactions plus Flink two‑phase… | ||
| Data governance in streaming at scale | Sebastián Ortega | 26m |
Letgo builds a company-wide data model using Avro schemas with field-level privacy tags that block or redact nonconforming events, and runs ingestion on Kafka… | ||
| No More Silos: Integrating Databases and Apache Kafka | Robin Moffatt | 27m |
Argues there are two concrete CDC approaches for getting database changes into Kafka: query-based CDC (JDBC connector) polls a timestamp or incrementing id, is… | ||
| Speech to text with Elasticsearch | Sophie Carboni, Lucian Precup | 37m |
Prototype turns speech into phonemes, indexes them in Elasticsearch and matches phoneme sequences against OpenOffice/CU dictionaries (English ~40k, French… | ||
| Scale up your job satisfaction, not your software | Bartosz Mikulski | 20m |
Presents concrete engineering practices to reduce wasted effort caused by lack of trust in data systems: define business requirements with real users, add… | ||
| Demystifying the Solr Operator | Houston Putman, Timothy Potter | 29m |
Solr Operator runs as a lightweight controller inside Kubernetes and manages Solr clouds via a SolrCloud CRD, templating StatefulSets, services, configmaps and… | ||
| Stateful Functions 3.0: Towards a Cross-Platform Stateful Serverless Stack | Tzu-Li (Gordon) Tai | 27m |
Stateful Functions 3.0 claims to provide Apache Flink-backed runtime that atomically couples messaging and distributed state so messages process as if exactly… | ||
| Change Data Streaming Patterns in Distributed Systems | Gunnar Morling, Hans-Peter Grahsl | 30m |
Shows concrete patterns and working code: CDC with Debezium + Kafka Connect reads Postgres transaction logs, routes outbox rows into per-aggregate Kafka… | ||
| Better, Faster, Stronger Streaming: Your First Dive into Flink SQL | Caito Scherr | 23m |
Flink SQL provides a concise way to build stateful streaming pipelines with the same semantics for bounded and unbounded data, claiming exactly-once guarantees… | ||
| The maintainability of open source code 20 years from today | Marcus Eagan | 29m |
A practical postmortem of Lucene/Solr maintenance with concrete wins and pain points: history of the 2010 merge and recent split, a documented pull request… | ||
| Powering Open Data Hub with Ray | Erik Erlandson | 21m |
Shows how Ray becomes an on-demand compute backend for Open Data Hub Jupyter notebooks by using ray.connect client-server instead of node-local ray.init and by… | ||
| The new OSS community: Enabling for advocates not customers | Andrea Griffiths | 29m |
Argues that shifting community efforts from sales funnels to listen with empathy and lightweight research (tweets, forum replies) creates measurable retention… | ||
| Commercial Models and Open Source: How Revenue Generation & Open Source Fit Together | Amanda Brock | 31m |
Argues that open source is a development and distribution approach not a business model, catalogs revenue patterns like support, subscription and open core… | ||
| Highly available and disaster ready Apache Solr | Anshum Gupta | 33m |
Explains concrete HA and DR features added to Apache Solr: incremental backups with checksummed index files, pluggable repository implementations (local, HDFS… | ||
| Faster Analytics for Fast Data with Apache Pinot and Flink SQL | Chinmay Soman | 39m |
Combines Apache Pinot for low latency OLAP storage with Flink SQL for streaming joins and repartitioning, ingesting Twitch events via Kafka and decorating from… | ||
| Vector Search: Ask Me Anything! | Dmitry Kan, Max Irwin | 01h 09m |
Practical Q&A on deploying vector search: benchmarks show a custom APU board can beat Elasticsearch for speed but requires heavy data prep; approximate nearest… | ||
| What's new in Apache Tika 2.0 -- we mean it this time! | Tim Allison | 21m |
Apache Tika 2.0 is shipping a modular rewrite: parsers split into parser-standard, extended and advanced, with 2.0.0-ALPHA in January and a beta in May… | ||
| Should you read Kafka as a stream or in batch? Should you even care? | Ido Nadler, Opher Dubrovsky | 32m |
A multi‑petabyte Kafka consumer is reworked from micro‑batches to hourly batch jobs using Spark on EMR, Airflow for scheduling, Kafka timestamp→offset reads… | ||
| Building a metadata ecosystem using the Hive Metastore | Neelesh Salian | 30m |
Stitch Fix runs a standalone Hive Metastore (2.3x) backed by MySQL RDS and builds a surrounding ecosystem: a Bumblebee REST proxy plus Python client that… | ||
| Be The Leader We All Need | Megan Byrd-Sanicki | 53m |
A concrete case shows a newcomer waits a year and forfeits 400 contributions lost before a volunteer mentor helps them become a top contributor; projects… | ||
| "Are You Sure?": blending product comparisons and recommendations with A.I. | Patrick John Chia | 33m |
Pipeline generates comparison-as-recommendations by first building dense product vectors from session co‑views and KNN retrieval, then refines candidates with… | ||
| From text search & recommendation to ads & online dating | Lester Solbakken | 28m |
Modern ANN libraries lose many relevant results when metadata filters are applied after retrieval; Vespa modifies HNSW to support eligibility lists so ANN… | ||
| Building Data-Intensive Teams | Elias Nema | 35m |
Project recommends a workflow: validate hypotheses with a fast SQL prototype, collect real user feedback via controlled A/B tests, then harden infrastructure… | ||
| Scaling Search Clusters with Apache Solr and Kubernetes | Amrit Sarkar | 29m |
Shows how Kubernetes plus Apache Solr are extended to perform physical and logical autoscaling of search clusters, using real metrics (query QPS and ingestion… | ||
| Data engineering & DataOps: Ask Me Anything | Lars Albertsson, Ellen Friedman | 54m |
Built early video conferencing at Google and later a Spotify internal data platform that democratizes analytics, uses Hadoop-era patterns (immutable inputs… | ||
| Log Management: From grep to Full-Text Search and Back | Philipp Krenn | 28m |
Compares concrete tradeoffs: Elasticsearch uses inverted index tokenization and aggregations to make full‑text searches fast at the cost of CPU, disk and… | ||
| Retrofit Your Java App with a Reactive Flow Pipeline | Mary Grygleski, Fabio Tiriticco | 34m |
A working demo wires Kafka -> Akka Streams (Scala) -> custom RSocket connector -> Project Reactor (Java) -> Kafka, with a sources producer pushing ~1… | ||
| How to avoid columnar calamities: what no one told you about Apache Parquet | William Benton | 39m |
Parquet wins size and speed by columnar layout, run-length and dictionary encodings, and predicate pushdown; a cited example compresses 50M synthetic payments… | ||
| Simplifying upserts and deletes on Delta Lake tables | Prashanth Babu | 31m |
Delta Lake lets you run atomic MERGE with UPDATE/INSERT/DELETE using SQL or Python/Scala and shows demo runs that report affected row counts and restore prior… | ||
| Explaining model explainability | Isabel Zimmerman | 37m |
Explains practical explainability: uses a support vector classifier on the MIMIC MRSA dataset (demo uses 108 samples, ~90% accuracy) and runs explainers from… | ||
| Kubernetes and the dynamic world in the cloud | Frank Conrad | 31m |
Argues that scaling short and big beats always-on clusters: running an 8x larger cluster for 4 hours yields similar cost and delivers results 8x faster. Shows… | ||
| Event-Driven applications: Apache Kafka and Python | Francesco Tisiot | 45m |
Practical walkthrough of Apache Kafka with Python showing actual code: use kafka-python producer/consumer code to write JSON/Avro bytes, create topics with… | ||
| Kubernetes & real-time analytics: How to connect these worlds with Apache Flink | Albert Lewandowski | 26m |
Compares running Apache Flink on Kubernetes via Lyft/GCP operators, the official deployment, Ververica Platform GUI, or bespoke scripts, and gives concrete… | ||
| Observability with Prometheus and beyond | Myrle Krantz | 27m |
Overview of a concrete stack: Prometheus for metrics with pull-based scraping, counters/gauges/histograms and PromQL for queries; single instances have handled… | ||
| Building Observable Streaming Systems with OpenTelemetry | Ricardo Ferreira | 30m |
Shows how to get end to end traces across Apache Pulsar by wiring OpenTelemetry into a Java Spring Boot producer and a Go consumer and proves it works. Uses… | ||
| Observability for data pipelines with OpenLineage | Julien Le Dem | 41m |
OpenLineage defines a JSON schema and start/complete events with a unique run id to capture runtime metadata so systems expose job runs, input and output… | ||
| Comparing apples to apple trees: Injecting domain knowledge to similarity searches | Yizhar Toren | 11m |
Claim: product-level similarity improves by injecting known image→product links into image similarity and aggregating up to products. Implementation overlays… | ||
| Text categorization with Apache Lucene | Lucian Precup | 12m |
Uses an Apache Lucene index itself as the trained model to do text categorization with no external ML libraries, indexing BBC dataset 2,225 articles into five… | ||
| Beyond Artificial Intelligence for Search | Lucian Precup | 09m |
Claims that for structured e‑commerce data a hand‑tuned search configuration — using boosts and function_scores, domain taxonomy and synonyms, redirects and… | ||
| Kickstart your Kafka with Faker Data | Francesco Tisiot | 11m |
Shows how to generate continuous test events and push them into Kafka using Python and the Faker library with a kafka producer client, including localization… | ||
| 5 min to discover an open source enterprise search solution | Cédric Ulmer | 07m |
Datafari claims a working community install from scratch in minutes via docker pull datafari/datafari-ce, then follow documented commands — the demo says… | ||
| Exploring the alchemy of Streaming and Solr cloud | Atita Arora | 10m |
E-commerce batch reindex that took 4+ hours becomes near‑real‑time by replacing twice‑daily Solr importer with Kafka streaming, Redis for nonsearch fields, and… | ||
| From Tables to Documents—Changing Your Database Mindset | Lauren Schaefer | 38m |
Compares MySQL and MongoDB with Python code and real examples, claiming four concrete gains: scale horizontally by sharding (cheaper, no downtime), query… | ||
| Building a new Big Data distribution based on Kubernetes with a twist! | Lars Francke, Sönke Liebau | 41m |
Claims an open source Big Data distribution that runs Kafka Spark NiFi HDFS on a Kubernetes control plane while replacing container runtimes with a Rust… | ||
| There is no such thing as bias-free tech. How models shape our world. | Daniel Guagnin | 20m |
Argues that “bias” is not just bad data but assumptions built into models that become invisible constraints in software and hardware, so unbiased tech is… | ||
| How to measure Diversity of Search Results | Eric Pugh | 32m |
Provides a concrete, repeatable measurement: compute Shannon entropy over result-source categories to score diversity, implemented as a notebook (Apache… | ||
| Two clicks to cluster and block thousands of spammers | Juan De Dios Santos Rivera | 35m |
LOVOO uses unsupervised models to let moderators two clicks cluster and act on user batches; the platform stitches an ML model service into an Antispam system… | ||
| The Acceleration of Cloud Engineering and AI: The Future of Work is Now | Grace Francisco | 41m |
Claim: cloud engineering treats infrastructure and apps as one software practice and it already drives hiring (AI/ML jobs forecast +71% in five years) and… | ||
| nothing matches | ||