Big Data Conference Europe 2023
2023-11-22..2023-11-24
Vilnius, Lithuania
[Big Data Conference Europe 2019, Big Data Conference Europe 2022, Big Data Conference Europe 2023, Big Data Conference Europe 2025]
who it's for
data engineers, data platform engineers, machine learning engineers, analytics engineers, data architects, data scientists
topics
vector databases; billion-scale vector search; Apache Spark application development; BigQuery storage best practices; metadata automation for data mesh; serverless data platform on Azure; Apache Beam introduction; real-time OLAP and stream processing; multi-cloud data platform patterns; FerretDB MongoDB to Postgres migration; scalable MLOps reference architectures; privacy-enhancing federated learning
| title | author | length |
|---|---|---|
| Towards Large-scale Speech Analytics Systems | Antonio Zarauz Moreno, Mateo Alvarez Calvo | 24m |
Claims a production system that audits telemarketing sales at scale by combining Whisper transcripts, voice activity detection, diarization and LLMs with… | ||
| Why Do You Need an Explainable Ai? | Sohini Pattanayak | 35m |
Argues that lack of transparency causes real legal and bias failures and that explainability is practical not philosophical, using LIME, SHAP and… | ||
| Sparking Success: Unveiling the Journey of Apache Spark Application Development | Pasha Finkelshteyn | 46m |
Builds a Spark pipeline that reads par files from the mov lens export, extracts year and explodes pipe‑delimited genres, groups and counts movies per year, and… | ||
| Getting Started with Apache Airflow - Building Your First Workflow | Ricardo Sueiras | 36m |
Clear, concrete outcome: a runnable DAG that uploads jokes to S3 and MySQL on a daily schedule, demonstrated successfully in Airflow 2.7.2 running via a local… | ||
| The Challenges of Making Vector Search Billion-scale | Kacper Łukawski | 34m |
Claims concrete tradeoffs and engineering wins: Qdrant uses HNSW with segmenting and memory-mapped files to avoid full in‑RAM graphs, demonstrates 100… | ||
| A 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet | Jay Alexander Clifford | 46m |
Claims concrete performance: using Arrow Flight, DataFusion, Apache Arrow and Parquet 87% smaller yields real speedups — a Spark round trip drops serialization… | ||
| Green Must Be Convenient | Roberto Freato | 37m |
Practical claims and live tools determine value here: the session shows Azure SQL Database managing vast JSON objects to bridge data lakes and relational apps… | ||
| Building Real-time Travel Alerts | Timothy J Spann | 53m |
Builds a real pipeline that ingests global GTFS‑RT and RSS, converts protobuf and CSV to JSON, pushes into Apache NiFi, then to Apache Kafka and streaming SQL… | ||
| AI Prompt Engineering to Customer Engagement: the Evolution of Effective Content | Atiba de Souza | 28m |
Explains how prompt engineering and an ideal customer profile turn vague AI output into repeatable production: ChatGPT custom instructions and custom GPTs… | ||
| Active-Active Machine Learning Infrastructures For Training And Prediction | Tim Frey | 36m |
Claims an active-active ML pipeline that does simultaneous training and prediction with streaming data and hot swaps models into production. Uses a Kafka… | ||
| Getting Started With Vector Databases | Zain Hasan | 41m |
Explains that vector databases store vectors alongside originals so semantic similarity returns meaning not keywords and scales to billions of documents; demos… | ||
| Don’t Worry About BigQuery PII | Karim Wadie | 37m |
Claims an automated, serverless pipeline that finds and protects PII across BigQuery at telecom scale and is open source. It uses Cloud DLP, BigQuery… | ||
| DataOps: Much More Than DevOps for Data | Sascha Giese | 30m |
Claims that apply DevOps lessons to data and that DataOps is about processes and mindset not database tinkering. Uses concrete examples: CPU metrics sampled… | ||
| AI Slides - Generating Slide Decks with AI | Thomas Endres, Jonas Mayer, Markus Schuttler | 44m |
Claims that AI produces complete slide decks and delivers them with Deepfake Avatars and a demo site are backed by specifics: text generation via GPT prompts… | ||
| Designing the Future: Stable Diffusion & the Rise of AI-Driven SaaS | Jeff Fan | 36m |
Demo builds a working AI‑SaaS: a three‑tier app called Story Forge that generates children’s illustrations with Stable Diffusion hosted on DigitalOcean +… | ||
| Using Modern Application Principals to Automate Your Apache Airflow Data Pipelines | Ricardo Sueiras | 35m |
Claims a reproducible Airflow pipeline workflow using CI/CD to build versioned container images and git-sync sidecar or shared volumes for DAG delivery… | ||
| Best Practices for Storing Data in BigQuery | Angelika Postaremczak | 33m |
Claim: changing table structure slashes query bytes without touching SQL — examples show a query going from 3.7 GB → 11 MB and another from 3.5 GB → 200 KB by… | ||
| Big Data and Small Data Everywhere, All at Once | Daniel Smith | 30m |
A large-scale data profiler gets ported to run entirely in the browser and it actually handles realistic loads: a client dataset of 6.5 billion records… | ||
| Metadata Automation in the Data Mesh | Sebastian Herold | 37m |
Zalando faces 50 million customers, ~150 data scientists, 500 AWS accounts and thousands of services and builds a unified metadata platform plus a data product… | ||
| Current Methods for Building Data Pipelines | Antonio Murgia, Bongani Shongwe, Elena Lazovik, Ravi Bhatt, Yingjun Wu | 01h 01m |
Panelists compare concrete stacks and tradeoffs: lots of orgs still ingest via SFTP into object stores, orchestrate with Airflow + Spark + DBT on object… | ||
| Why Most Data Projects Fail and How to Avoid It | Jesse Anderson | 31m |
Claims that 85% of data projects fail and only 15% succeed backed by Gartner numbers lead to concrete prescriptions: prefer Snowflake/Databricks-aware… | ||
| Streams to the River: Powering an Online XGBoost Classifier with Kafka | Mauro Mariniello | 41m |
Built an Adaptive XGBoost that learns from a Kafka stream using a fixed-size buffer, an EDWIN drift detector, and three ensemble strategies (push, replace… | ||
| Digital Dystopia or Artistic Utopia? The AI Creative Quandary | Matthew Blakemore | 41m |
Claims a patented multi-territory age rating solution and a delivered multimodal AI for detecting issues built with Amazon and University of Bath, and shows a… | ||
| Four Years of Data Mesh in Practice | Emily Gorcenski | 42m |
Claims four years of production experience and a concrete reference implementation: a CLI that provisions a data product in about 25–30 minutes, multicloud… | ||
| Implement a Serverless Data Platform with Microsoft Azure | Roberto Freato | 43m |
Shows an implemented Azure pipeline that actually moves events from Event Hubs into DataLake Storage, uses Stream Analytics to aggregate into partitioned… | ||
| What are Fish? The Ethics of Rules | Steve Upton | 53m |
Connectionist systems like neural networks and large language models embed decision rules across weights so connectionist systems hide rules and explainability… | ||
| Hit the (ML) Road(map) Jack | Yael Daihes | 44m |
A concrete framework emerges: generate a large list of one model task ideas, score each by five concrete metrics (model development, performance measurement… | ||
| Navigating Data Residency Challenges | Nicolas Fränkel | 31m |
Concrete options for routing and enforcing residency appear: place routing logic in client, server, proxy or API6 gateway, or combine them to reduce errors… | ||
| Collect Data to Fine-Tune Your Large Language Model | Mauro Bennici | 39m |
Claims concrete tradeoffs: training GPT‑3 class models costs <$5 million training estimate and uses billions of parameters so fine‑tuning a small domain model… | ||
| Attacking (and Defending) Apache Kafka | Francesco Tisiot | 43m |
Clear, concrete attack scenarios and specific defenses: explains abusing consumer group membership to halve notifications, forcing rebalances from a single… | ||
| Data Acquisition: Overcoming Challenges | Artsiom Yudovin | 29m |
Practical account of a working pipeline that streams production changes into a lake: Basium Kafka Connect to Apache Kafka, Apache Spark structured streaming… | ||
| Rings of Security for Big Data in Cloud | Rachit Arora | 31m |
Claims concrete defenses: a managed Big Data cluster-as-a-service running a vetted Hadoop distribution with Ingress gateways fronting traffic, load balancers… | ||
| The Stuff You Don't Want to Write... Documentation | Kris van der Mast | 38m |
Argues that documentation must be written alongside code and baked into the definition of done so each two‑week sprint produces usable notes, and demonstrates… | ||
| Hamelin 77, Alignment and Control in an Ecosystem of Diverse AI Agents | Javier Ideami | 39m |
A short film called Hamelin 77 claims to probe AI alignment while actually building tools: it uses NeRF instant NGP to reconstruct a historical interior with… | ||
| A Gentle Introduction to Apache Beam | David Regalado | 26m |
Claims that Apache Beam unifies batch and streaming with a single programming model and portable runners, demonstrates SDKs for Python Java Go SQL and Scala… | ||
| Real-time Customer Engagement in Gaming Using Kafka and Flink | Volker Janz | 40m |
Real pipeline detail: a game tracks interaction events into Kafka and Flink, evaluates campaigns via a JavaScript filter broadcast through a low‑frequency CRM… | ||
| How Can We Apply Agile Principles and Practices in Analytics? | Agnė Kelminskienė, Gabrielė Adomonytė, Vaiva Mikelevičienė | 43m |
Claims that matter: analytics moves faster when analysts joining engineering scrums and when one engineer is dedicated data engineer for data science so the… | ||
| Accelerating Query Performance in CrateDB | Marija Selakovic | 38m |
Shows concrete tactics and measured wins: use EXPLAIN ANALYZE to see Lucene-time breakdowns, prefer partition filters to avoid hitting 250 shards, and replace… | ||
| Fast, Faster, Fastest: Object Storage, Cloud Block Storage, and SSD | Denys Kondratenko | 30m |
Claim: for ClickHouse workloads storage often isn’t the bottleneck — local NVMe, EBS and S3 give similar query latency because ClickHouse does heavy… | ||
| Lessons Learned from the GCP Data Migration | Damian Filonowicz | 41m |
PAYBACK moves terabytes into BigQuery and Cloud Storage using Cloud Dataflow (Apache Beam) with on‑the‑fly pseudonymization via Cloud DLP tokenization, then… | ||
| Challenges Faced and Lessons Learnt While Building a Multi-cloud Data Platform | Antonio Murgia | 34m |
Transforms an AWS‑centric data platform into a multicloud setup, implements a Hive proxy catalog that mimics Glue and a custom Spark catalog, and runs… | ||
| CDC: From Zero to Hero | Andrei Tserakhau | 24m |
Compares logical replication vs CDC and shows a concrete pipeline built inside double Cloud transfer that reads DB write‑ahead logs, streams events to Kafka… | ||
| Moving MongoDB Workloads to Postgres with FerretDB | Peter Farkas | 32m |
FerretDB claims it lets MongoDB applications run unchanged on Postgres by translating the MongoDB wire protocol into SQL and shipping under Apache 2.0, turning… | ||
| Analytics in 2030 | Asta Bagdonavičienė, Diana Gold | 41m |
Predicts specific near‑future tech and concrete pilots: Quantum computing: a commercial Finnish unit sells for €1M to universities but isn’t production ready… | ||
| Peer Beneath the Surface of the Earth | Maciej Marek | 41m |
Claims a new radar collects 50,000 points per second (not three) and Widmo builds a pipeline that turns one-row profiles into more than one million generated… | ||
| Search: A New Era | David Pilato | 38m |
Elasticsearch now stores and searches dense embeddings, supports 4,000 dimensions (tradeoff: memory and slower indexing), and uses HNSW + brute force switching… | ||
| Real time OLAP and Stream Processing Friends or Foes | Yingjun Wu | 34m |
Claim: real-time analytics needs two different engine types because of a freshness vs latency vs cost tradeoff; practical demos use CDC plus SQL streaming… | ||
| Data Mesh: Challenges and Opportunities | Amy Raygada, Arne Laponin, Maciej Marek, Matteo Pelati, Peter Farkas | 01h 00m |
Panelists report concrete outcomes not slogans: teams break a monolithic warehouse into data products with automated platform checks, migrate on‑prem to cloud… | ||
| Statistics Estonia Chose Denodo | Tauno Tamm, Laurent Chailley | 25m |
Statistics Estonia replaces underlying databases and keeps all queries working by adding a logical layer built on Denodo that virtualizes sources and serves… | ||
| Deep Dive - Architecting for Million Operations Per Second! | Ravi Bhatt | 37m |
Design proves how to reach millions ops/sec by building a pipeline that generates synthetic CSVs with Dataflow, stages files in GCS, and coordinates write/read… | ||
| Big Data With Open Source Solutions | Igor Khrol | 32m |
Automattic runs a Hadoop HDFS backed data lake on bare metal with Spark for ETL, Trino for low‑latency SQL, Yarn as the cluster resource manager, Airflow… | ||
| Unlocking Real-time Insights: Enhancing Your Databases With Stream Processing | Yingjun Wu | 48m |
Claims that a streaming SQL database (RisingWave, open‑sourced April 2022 with ~5k stars) lets applications use Change Data Capture into materialized views so… | ||
| Challenges of Building Blazing Fast Data APIs | Matteo Pelati | 39m |
Claims a working system called Dozer that turns CDC feeds into low‑latency, customer‑facing Data APIs using a single binary written in Rust and a realtime… | ||
| Querying Live Data With LLM App | Bobur Umurzokov | 38m |
Claims a working pipeline that builds an AI app in 30 lines of code using OpenAI API and Pathway Python libraries, ingests Dropbox and streaming sources, does… | ||
| Unleashing the Power of Unstructured Data | Ani Megerdoumian | 32m |
Holmes starts as a proof of concept and becomes a global real time search engine for ING wholesale banking that indexes text PDFs web pages and credit packs… | ||
| Automating Data Analytics: Collect, Transform and Deploy - Lead With Change | Aleksejs Vesjolijs | 33m |
Claims full automation across the pipeline from pre-code checks to release management and monitoring, implemented with GitHub Actions, Snowflake, Airflow on… | ||
| Unified Customer Data Management | Valentina Tortolini | 29m |
Claims a warehouse-native customer data modeling pipeline that ingests behavioral and CRM data, resolves identity via hashed emails plus optional… | ||
| A “Moving SLO” For Machine Learning and AI Services | Emily Gorcenski | 38m |
Proposal reframes SLOs as residual error as SLO by running a predictor corrector (Kalman‑family) over model inputs and observable outputs and averaging the… | ||
| Enriching Postal Addresses With Elastic Stack | David Pilato | 35m |
Live coding shows how Elasticsearch, Logstash, Kibana and Filebeat turn messy CSVs into usable location data: a CSV ingest pipeline parses street number, name… | ||
| Databases Can't Be Just Databases Anymore | David Gomes | 28m |
Claims that databases become platforms drive concrete changes: MongoDB turns Atlas into a cash generator (Atlas revenue rises from ~1% to ~60%), adds Atlas… | ||
| From Prototype to Production Building a Scalable MLOps Platform from a Reference Arch | Bas Geerdink | 41m |
A concrete reference architecture connects training pipelines, a model registry, a feature store and online scoring behind an API, with orchestration via… | ||
| Building an Event Analytics Pipeline with Kafka, ksqlDB, and Druid | Hellmar Becker | 37m |
Claims a working event analytics pipeline built on Kafka, ksqlDB, and Druid and demos it live, standing up filtering and streaming ETL on a laptop in under… | ||
| A Privacy Enhancing Federated Learning Framework for Multi-Party Distributed .. | Elena Lazovik | 28m |
Claim: keep raw data local and train global models instead of moving terabytes of energy data. Built a proof‑of‑concept federated learning platform using… | ||
| Building Recommendation Systems for Global E-commerce Marketplaces | Bongani Shongwe | 33m |
Adevinta builds centralized recommender and profiling services using Vertex AI and GCP (BigQuery, Spark, Cloud Run, Cloud Functions), migrating from on‑prem to… | ||
| 3-Step Path for Fixing Data Discoverability | Arne Laponin | 23m |
Claims that a catalog becomes useful only when it supports concrete features: searchable metadata, automated access requests, shallow snippets and masked PII… | ||
| Data Mesh: Kicking Off a Mentality Change in an Organisation | Amy Raygada | 36m |
Argues that Data Mesh requires organizational change not just infra and claims a measurable rollout: migration to cloud with GCP BigQuery dbt Looker Atlan… | ||
| nothing matches | ||