Flink Forward Global Virtual 2020
2020-10-19..2020-10-22
-- online --
| title | author | length |
|---|---|---|
| Keynote | Flink Ahead: What Comes After Unified Batch & Streaming? | Stephan Ewen | 37m |
Flink claims unified batch/streaming now handles petabytes per day with SQL, and community releases cut TPC-DS (10 TB on 20 machines) runtimes by about 3x… | ||
| Keynote | Flink Ahead 2.0: The Sequel | Konstantin Knauf | 34m |
Flink SQL powers continuous materialized views and Ververica Platform 2.3 ships an integrated editor, catalog, and autopilot for horizontal scaling. Live demos… | ||
| Keynote | Flink First: Designing Modern Trading Systems | Peter Chalif | 37m |
A Flink-first trading platform replaces a decades-old patchwork of tools with a unified stream backbone so every component is a Flink job, handling roughly 8… | ||
| Keynote | Real-Time Metrics at Fortnite Scale | Ricky Saltzer | 28m |
Metrics Royale extracts and aggregates arbitrary JSON fields with Apache Flink and Kafka, persisting partial two‑second aggregates into FoundationDB so… | ||
| Keynote | Data Warehouse, Data Lakes, What's Next? | Xiaowei Jiang | 38m |
Proposal: a unified HSAP storage called Hologres stores row and column formats, exposes Postgres-compatible SQL, and couples with Flink for realtime dimension… | ||
| Keynote | Stream Processing at LinkedIn: Then and Now | Renu Tewari | 35m |
LinkedIn runs stateful stream processing at massive scale: Kafka ingests over 11 trillion incoming messages per day and stream jobs process over 4 trillion… | ||
| Unified APIs for Batch and Stream Processing on Flink | Aljoscha Krettek | 31m |
Flink adds a new execution.runtime.mode toggle (streaming / batch / automatic) and modernizes time semantics so event time is the default and watermarks jump… | ||
| Cluster Serving: Distributed Model Inference using Apache Flink in Analytics Zoo | Dongjie Shi, Jiaming Song | 18m |
Cluster Serving runs on Apache Flink via Analytics Zoo and provides automatic distributed serving for TensorFlow, PyTorch, Caffe, BigDL and OpenVINO models… | ||
| Building a Self-Serve Stream Processing Platform with Flink | Manuela Wei, Ping Zou, Nick Hwang | 36m |
Intuit builds a self-serve platform that wires Beam and Flink to Argo and Kubernetes, auto-provisions GitHub/Jenkins/artifactory for each processor, and… | ||
| Flink SQL: Unified Streaming and batch for hive integration | Jingsong Lee, Rui Li | 36m |
Flink 1.11 adds Hive metastore integration so Kafka-streamed tables persist to HDFS-backed Hive tables with exactly-once guarantees via the streaming file sink… | ||
| Real-Time ML Applications at Scale with Flink and Beam on Kubernetes and GCP | Chethan U | 58m |
Rapido runs a production streaming platform that uses Flink on GKE to make near‑real‑time dispatch and ML scoring decisions, integrating Presto for analytics… | ||
| Handling frequent changes in state structure | Gaël Renoux | 16m |
A utility that forces an operator to reset when its state layout changes is built and running in production for DataDome using Flink and Scala, with code at… | ||
| Enriching your Data Stream Asynchronously with Apache Flink | Hari Rajaram, Jeremy Ber | 26m |
Shows how to replace blocking lookups with Flink’s async I O using RichAsyncFunction and demonstrates it running on Amazon Kinesis Data Analytics for Apache… | ||
| Data science at scale using Apache Flink: Dynamic model serving and real-time feature generation | Shashank Agarwal | 15m |
Describes a production platform called Mitra built on Apache Flink + Kafka + RocksDB that serves 500+ e‑commerce companies and performs real‑time model scoring… | ||
| Sweet Streams are Made of These: Applying Metrics-driven Development to Apache Flink | Caito Scherr | 21m |
Practical recipe for making Flink production-ready: build a tight metrics cycle using Flink’s REST API and reporters (Prometheus, DataDog) and pair with an… | ||
| Streaming, Fast and Slow: Mitigating Watermark Skew in Large, Stateful Jobs | Shahid Chohan | 15m |
Stripe builds a custom dual source that backfills from S3 and transparently hands off to Kafka, but large backfills stall due to watermark skew and ballooning… | ||
| Flink + Pulsar: The Path To Unified Batch and Streaming | Addison Higham | 30m |
Flink reads from Pulsar with native tiered storage and segments offloaded to S3, enabling parallel batch reads from closed segments and seamless transition to… | ||
| Introducing Tensorflow on Flink | Xuannan Su | 35m |
An implementation that runs TensorFlow (and PyTorch) inside Flink by exposing machine‑learning worker/PS roles as Flink operators and an application manager… | ||
| Hands-on Stateful Serverless Applications with Kubernetes and Stateful Functions | Seth Wiesman | 34m |
Shows building a real Stateful Functions app atop Apache Flink: Python and Go remote SDKs, protobuf Any messages, Kafka ingresses/egresses, MinIO checkpoints… | ||
| Growing a managed Flink streaming platform from scratch | Chen Qin | 40m |
Pinterest builds a managed Flink platform that onboards over 25 projects and productionizes 27+ jobs in H1 2020 by shipping a unified source (reads Kafka or… | ||
| Sampling and Modelling the World in Real-Time for Mobility | Alberto Romero | 38m |
A real-time platform ingests telematics at 10–100 Hz telematics and GPS at 1 Hz, fuses weather, road features and police accident records, and produces… | ||
| A Year in Flink - Flink SQL Live Coding | Nico Kruber | 31m |
A live Flink SQL demo rebuilds community stats by ingesting GitHub commits, PRs and three mailing‑list mbox archives into Kafka then querying them from the… | ||
| Metamodelization and Automatic Generation of ML Pipelines | David Massart | 15m |
Claims an Apache Flink + Kafka pipeline that serves PMML models delivered via Kafka and auto-generates the Flink job code for enrichment steps. Enrichments are… | ||
| Towards a Flink Autopilot in Ververica Platform | Fabian Paul | 19m |
Ververica Platform Autopilot automatically reconfigures Flink jobs by combining operator idle time ms/sec with source Kafka lag to compute exact parallelism… | ||
| FlinkNDB : Skyrocketing Stateful Capabilities of Apache Flink | Sruthi Sree Kumar, Muhammad Haseeb Asif | 24m |
FlinkNDB replaces embedded RocksDB with an external state backend built on MySQL Cluster Engine and demonstrates stateful scaling to hundreds of terabytes… | ||
| How Nextdoor improves their customer experience in real time with Apache Flink | Deepthi Mohan, Eugene Brechko | 16m |
Built an Interaction Store that consumes Kinesis Data Streams, processes with Apache Flink on Kinesis Data Analytics, and writes to DynamoDB; state holds… | ||
| Splunk Data Stream Processor | Jamie Grier | 21m |
Splunk’s Data Stream Processor routes ingest through Apache Pulsar into Flink jobs compiled from SPL via an AST type checker and planner that emits a Flink job… | ||
| End-to-end Flink SQL with Schema Registry, Kafka & Kudu | Gyula Fora, Balazs Varga | 31m |
An end-to-end Flink SQL pipeline reads Avro records from Kafka via the Schema Registry catalog, enriches by joining on sensor_id to a Kudu table exposed… | ||
| Building Unified Streaming Platform at Uber | Zhenqiu Huang, Venkata Sanath Muppalla | 38m |
Uber runs 2,500 Flink pipelines handling more than 6 trillion messages per day and builds a unified deployment layer that manages both Streaming SQL and native… | ||
| When your stream doesn't stream very well | Niels Basjes | 29m |
Apache Flink + Kafka jobs stall on test or batchy streams because windows wait for watermarks and sinks buffer small volumes; common failure modes are slow… | ||
| Data processing at Spotify - Why portability matters | Julien Tournay, Filipe Regadas | 36m |
Spotify uses Scio, a Scala API on Apache Beam, to run the same job on Dataflow -> Flink or Spark or Hadoop with no code change. Migration from a 2,500-node… | ||
| News from Flink’s engine room: “Full steam ahead” | Till Rohrmann | 44m |
Flink adds a pipeline region scheduler that makes pipeline regions the atomic unit for scheduling and failover and avoids deadlocks when resources are… | ||
| Using the Mm FLaNK Stack for Edge AI (Flink, NiFi, Kafka, Kudu) | Timothy J Spann | 32m |
A working edge pipeline that reads desk sensors with MiNiFi -> NiFi -> Kafka, validates and writes Avro with a central schema registry, and runs a continuous… | ||
| Change Data Capture and Processing with Flink SQL | Jark Wu, Qingsheng Ren | 42m |
Flink SQL 1.11 implements FLIP-95 so sources can emit insert/update/delete changelog and the pipeline reads snapshots then binlogs; a demo wires MySQL/Postgres… | ||
| Flink @WORKDAY: scaling of ingestion data pipelines to S3 | Leire Fernandez de Retana, Enrico Agnoli | 31m |
Flink jobs scale from 4k+ buckets and tiny per-bucket streams to single-bucket batch loads of 150M+ messages by replacing the default sink with a two-phase S3… | ||
| A journey of scaling high-volume low-latency stateful processing | Arti Pande | 34m |
Pipeline targets 1M TPS peak and proves feasibility by hitting 3.5M TPS on the stateful dedupe operator using Apache Flink with Kafka as source after… | ||
| Load Imbalance | Dmitry Voronov | 09m |
Resolving data spikes imbalance with additional Sub-Keys and Sharded MapState
Motivation:
Problem 1:
MCAS team uses Flink Streaming Framework for Real-Time… | ||
| Building an End-to-End Analytics Pipeline with PyFlink | Marta Paes | 25m |
A runnable pipeline that reads mailing‑list rows via the Postgres CDC connector, classifies each message with an LDA model trained locally using gensim and… | ||
| On-demand low-latency feature generation at Hyperconnect | Jaehyeuk Oh, Jeongmin Kim | 15m |
Hyperconnect builds a Flink-based real-time feature store (three layers: User Event Store, Feature Generator, Feature Consumer) that serves on-demand features… | ||
| Disco: Efficient Distributed Window Aggregation | Lawrence Benson | 19m |
Disco implements distributed window aggregation by slicing streams, pre-aggregating at leaf and intermediate nodes, and merging intermediate results at the… | ||
| Interactive Session - Live Code Contribution to Apache Flink | Seth Wiesman | 40m |
Flink repo with over a quarter billion lines gets a concrete feature: add an executeAndCollect API that returns a closable iterator and a bounded collect… | ||
| Tale of Stateful Stream to Stream Processing | Ajit Koti | 31m |
Netflix migrates its end-to-end batch payment workflow for 180M+ members across 190 countries to a multi-step Apache Flink streaming pipeline, rewriting big… | ||
| How We're Democratizing Stream Processing at Zillow | KELLY SMITH | 39m |
Zillow replaces slow ticketed builds with a self‑service platform that provisions Flink jobs, Kafka topics and permissions in minutes using Kubernetes as the… | ||
| CEP platform handling millions of users - lessons from 3 years in production | Krzysztof Zarzycki | 34m |
A Flink-based CEP platform ingests Kafka streams from NiFi and on-prem Hadoop and now serves over 10 million subscribers, ~500k interactions/sec and billions… | ||
| Flink's New Dockerfile: One File to Rule Them All | Andrey Zagrebin | 15m |
Flink replaces many scattered Dockerfiles with a single official Dockerfile that builds one image for jobs and clusters and explains how to deploy to local… | ||
| Production Ready Streaming SQL with Ververica Platform | Fabian Hueske | 20m |
Ververica Platform adds Flink SQL support and ships a web-based SQL editor with catalog browsing, auto-complete, interactive preview and schema inference. It… | ||
| Flink powers Houzz near-realtime pipeline & Kubernetes logging observability | Menglei Sun | 20m |
Within a year Apache Flink becomes Houzz’s near‑real‑time engine, running 15+ production jobs on Kubernetes and processing 10+ TB/day into Kafka topics; an… | ||
| Alert detection in IoT using broadcast state. | Lasse Nedergaard | 17m |
At Trackunit we collect and monitor sensor data from construction machinery around the globe to bring valuable machine insights to our customers.
We take… | ||
| Using Flink for Versatile Feature Engineering Pipelines | Michal Ciesielczyk | 20m |
Deep.BI builds a Flink-based feature-engineering framework that turns raw event streams into reusable features and live predictions with under one second… | ||
| Zero-downtime upgrades of Flink applications | Lakshmi Rao | 16m |
Lyft moves stateful Flink jobs from EC2 to Kubernetes and implements Flink Kubernetes Operator-driven blue‑green upgrades that create a new cluster, take a… | ||
| Powering OLAP at Uber using Apache Flink | Chinmay Soman, Girish Baliga | 15m |
Apache Flink converts user SQL into production streaming jobs that read Kafka, apply transformations via AthenaX, and write Kafka output that Apache Pinot… | ||
| Getting Started with Flink for Small Teams w/ RabbitMQ and Kubernetes | Austin Crawley-Edwards | 17m |
Walks through a concrete small-company pipeline: Twitter ingestion into a Flink job that buckets hourly analytics and bulk indexes to Elasticsearch, then… | ||
| Realtime bot detection with Flink | Pradeep Nethagani | 15m |
Yelp builds a Flink-backed Bot Signal Platform that processes 2.5 billion events per day and holds about 1 TB state to classify users as bot or not in near… | ||
| Feature Stores: Powering Machine Learning Infrastructure with Apache Flink | Simba Khadder | 15m |
Built a production feature store on Apache Flink feeding 10s of millions of users and used for personalization at ~100M MAU, with pipelines on Flink + Pulsar +… | ||
| Interview and AMA with the creators of Stateful Functions | Seth Wiesman, Stephan Ewen, Robert Metzger, Igal Shilman | 42m |
Creators of Apache Flink’s Stateful Functions describe a runtime that exposes virtual actors with message passing and colocated durable state, using Flink… | ||
| nothing matches | ||