Flink Forward Global 2021
| title | author | length |
|---|---|---|
| Apache Flink in the Cloud-native Era | Stephan Ewen | 36m |
Apache Flink is moving toward cloud-native operations with concrete features: buffer debloating reduces in-flight data so checkpoints become snappier, aiming… | ||
| Flink architecture and automation in Intesa Sanpaolo | giovani l'auspicio, aim raffaele saginaw, andy fenby, lio verdi | 32m |
Intesa Sanpaolo runs Apache Flink on containerized OpenShift clusters with cross‑region storage replicas and a plan for active‑active cloud migration… | ||
| Sharing what we love: The Apache Flink story at Pinterest | Chen Qin | 25m |
Pinterest replaces Kafka Streams with Apache Flink to run large-scale stream apps that power features for 478 million users, claiming reduced latency from… | ||
| Expert Panel: Current & Future State of the Art in Real-Time Data Infrastructure | 47m | |
Panel contrasts two adoption paths: pumping fresher analytics into warehouses versus powering low-latency, stream-driven microservices for feature extraction… | ||
| Dataflow Engines for Executing Cloud Applications: a Maslow's Hammer or Natural Outcome? | Asterios Katsifodimos | 20m |
Argues that large-scale stateful serverless workloads map to dataflow graphs and run on existing engines, shows concrete examples (shopping cart, payments)… | ||
| Only SQL: Empower data analysts end-to-end with Flink SQL | Martijn Visser | 25m |
Demonstrates that non-programmers can build end-to-end streaming workflows using only Flink SQL: reads clickstream and speech data from Kafka (Confluent REST… | ||
| A Year in Flink: The Most Important Changes in Recent Versions | Till Rohrmann | 37m |
The talk reviews the last year of Apache Flink development, covering three major releases that added about 7,000 commits and 3,000 fixed issues. It describes… | ||
| Introduction to Flink in 30 minutes | David Anderson, Johannes Moser | 27m |
A working mobile data pipeline is built and demoed: Kafka sources feed Flink in both batch and streaming modes, Flink SQL performs a temporal join (system_time… | ||
| Batching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors | Felix Gessert, Wolfram Wingerath | 27m |
Claims a production pipeline that scales from 1 to 100+ million users in a single month by replacing ETL/Athena batch jobs with a continuous analytics stack… | ||
| Real Time Fraud Detection with Stateful Functions | Levin von Hollen, Jan Brusch, Joscha Cepok | 29m |
Real‑time fraud and bot detection for webshops implemented with Flink Stateful Functions, Kafka/MSK, Redis and Postgres, running on EKS; HTTP logs plus… | ||
| XStream: Stream Processing Platform at Facebook | Aniket Mokashi, Shuyi Chen | 23m |
XStream consolidates Scribe, Puma, Stylus, Laser and Swift into a single managed platform built on Stylus, uses a vectorized evaluation engine from Velox, and… | ||
| Networks! project - real-time analytics that controls 50% of mobile network in Poland | Maciek Bryński, Rafał Malanij | 27m |
Real-time ingestion and analytics platform processes 2.2 billion messages a day from 750 Kafka topics to compute more than 5000 KPIs and 1500 SQL-defined… | ||
| A Debuggers Guide to Apache Flink Streaming Applications | Alexander Fedorov | 30m |
Step-by-step checklist for debugging Flink streaming: use metrics (failed checkpoints, consumer lag, throughput, checkpoint duration/size, barrier alignment… | ||
| Interactive Session: Flink on Autopilot with Splunk | Maximilian Michels | 38m |
Splunk describes a Fully Managed Autoscaled and Elastic Stream Processing Service that manages ten thousands of Flink jobs by extracting a control plane called… | ||
| Building payment processing engine with Stateful Functions and Spring Boot | Tymur Yarosh | 16m |
Proof of concept payment engine implements account and transaction as separate namespaces, deployed as two Spring Boot remote SDK modules; account stores… | ||
| Using Flink SQL on Ververica Platform | Ingo Bürk | 21m |
Ververica Platform runs Flink SQL with pre-packaged connectors (Kafka, datagen, formats like Avro/JSON) and lets teams upload custom connector JARs (example… | ||
| Building an open-source ML feature store with Apache Flink | Roman Grebennikov | 31m |
An open source Flink-based framework called Featury implements a protobuf change log written to S3 that is readable both for low-latency online inference and… | ||
| From concept to production; the fastest way to develop & deploy Apache Flink jobs | Danny Cranmer, Steffen Hausmann | 40m |
Live demo shows Apache Zeppelin notebook tied to Kinesis Data Analytics for Flink, reading from Kinesis Data Streams and Kafka, joining with S3 lookups… | ||
| Getting into Low-Latency Gears with Apache Flink | Jun Qin, Nico Kruber | 32m |
Practical checklist with measured gains: increasing parallelism cuts 99th‑percentile latency from ~3s to 650ms by reducing per-task CPU load; switching from… | ||
| Keeping Redditors safe in real-time with Flink Stateful Functions | Frédérique Mittelstaedt | 31m |
Reddit builds a real-time safety platform called Snooron that uses Flink Stateful Functions with Kafka and Kubernetes to detect and act on harmful content at… | ||
| Change Data Streaming Patterns in Distributed Systems | Gunnar Morling, Hans-Peter Grahsl | 41m |
Shows how Debezium plus Kafka Connect or the embedded engine capture DB transaction logs to implement three practical patterns: reliable event exchange via an… | ||
| Interactive Session: Running Flink SQL jobs in production | Gyula Fora, Erik Beebe, Matyas Orhidi, Marton Balassi | 36m |
Demo shows a Flink-based product wiring Kafka to external stores (Kudu, Hive) with an auto-detect Kafka table wizard and a Faker connector to generate data… | ||
| PyFlink: Building Flink applications with Python language | Dian Fu, Xingbo Huang | 35m |
PyFlink adds Python Table API/SQL and DataStream API support enabling Python UDF, UDTF, UDAF, Pandas UDF/UDAF and uses Py4J to map Python APIs to JVM jobs… | ||
| Real-time bot mitigation with machine learning in Flink | Gaël Renoux, Konstantina Kentoudi | 30m |
Datadome runs real-time bot blocking on Apache Flink over ~5 billion requests/day and produces per-IP verdicts with a business target of <0.01% false… | ||
| Interactive Session: Optimising data streaming pipelines on Flink and Kafka | Olena Babenko, Gyula Fora, Francesco Tisiot, Lorna Jane Mitchell | 38m |
Practical tips for running Flink with Kafka: watch for skewed partitions—one hot partition can break both Kafka and Flink; fix producer keying or rebalance… | ||
| Integrating Flink into Your Ecosystem - How to Build a Flink Connector from Scratch | Arvid Heise | 34m |
A concrete guide to building a Flink connector from scratch that both reads from and writes to real systems, providing working source and sink implementations… | ||
| Stateful Functions: Flexible like a Microservice, Consistent like a Monolith | Seth Wiesman | 24m |
Claims a runtime built on Apache Flink provides exactly-once state semantics and eliminates pay-for-wait time by inverting messaging so stateful storage routes… | ||
| Data Pipeline Lifecycle: SQL Everywhere | olenna babenko | 31m |
SQL from Presto is reused verbatim in Flink SQL to turn a historical analyst query into a streaming pipeline that joins a Kafka four-message toy dataset with… | ||
| Sources, Sinks, and Operators: A Performance Deep Dive | Brent Davis | 36m |
Splunk describes scaling a Flink job to 20 GB/s (~1.7 PB/day) on a GCP Kubernetes fleet of 200+ n1-standard-32 VMs and thousands of task threads, and shows… | ||
| Interactive Session: Beam and Beam on Flink general roadmap discussion and AMA | Kyle Weaver, Kenneth Knowles, Yifen Mai | 39m |
Interactive AMA covers the Beam-on-Flink roadmap with concrete claims: cross-language portability now lets Java connectors be used from Python and Go, the Go… | ||
| Big Data Processing : Where do we go next ? | Roshan Naik | 32m |
Claims that Streaming 2.0 can replace separate batch and streaming stacks by merging real-time and offline streaming, adding features like kappa+ backfill… | ||
| Polyglot Actor Model on Flink Stateful Functions | Timothy Bess | 38m |
Compares classic actor model (lightweight processes, cheap millions of actors, low-latency distribution) with Flink Stateful Functions that store durable… | ||
| The easy way to build and run Flink SQL jobs using Cloudera SQL Stream Builder | Kenny Gorman | 32m |
Demo shows SQL Stream Builder running on Apache Flink inside Cloudera, using DDL to create tables from a faker connector and Kafka topics, auto-detecting… | ||
| NRTG: Fast, Systematic & Scalable Flink Application Development at Pinterest | Divye Kapoor, Heng Zhang | 30m |
Pinterest describes a Flink framework that compiles a single dataflow class into optimized Flink graphs, adds Kafka sources, async RPCs, and backfill support… | ||
| Demystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about? | David Moravek | 30m |
Explains concrete deployment choices for Apache Flink: application mode bundles job and cluster to avoid jar distribution, session clusters use a two-step… | ||
| Jump start large window computations with Hybrid Source | Thomas Weise | 30m |
Bootstrapping large-window state by replaying weeks of data is solved by reading historical files from Iceberg + Kafka then switching to live Kafka without job… | ||
| Fitting Stateful Functions into a streaming platform | dylan meisner | 27m |
Nordstrom runs a Kafka-based streaming platform handling over 5,000 distinct resource changes per day and uses Prometheus for telemetry. Flink Stateful… | ||
| Building Presto SQL experience on Flink | Peter Huang, Zhongting Hu | 34m |
Adds Presto SQL as a dialect on Flink inside AthenaX by translating Presto ASTs into Calcite/Flink plans and by optionally using Presto codegen; implements… | ||
| ML Inference at scale, with Apache Beam and the Flink runner: Easy as learning your 5 times tables. | REZA ROKNI, Robert Crowe | 39m |
Builds beam pipelines that run TensorFlow models with the Flink runner using the RunInference transform from tfx bsl, loading SavedModel from GCS in local mode… | ||
| Fault Tree Analysis Applied to Apache Flink | Andrey Falko | 35m |
Applies classic Fault Tree Analysis to Apache Flink, models Kafka, ZooKeeper, Kubernetes, S3 and checkpoints, and ships an SLO calculator plus step‑by‑step… | ||
| Storing State Forever: Why It Can Be Good For Your Analytics | Yaroslav Tkachenko | 32m |
Shows a concrete Flink 1.12 implementation that keeps key state forever: a non-temporal union join that ingests tens of billions of CDC messages from Kafka… | ||
| Paper Moon: A Comparison of Stateful Functions and Pulsar Functions | Jowanza Joseph Finicity | 36m |
Compares Flink Stateful Functions vs Pulsar Functions by concrete tradeoffs: Stateful Functions uses Flink for durable state with checkpointing/snapshotting… | ||
| Interactive Session: AMA | Pulsar and Flink for Unified Data Processing | cj gal, addison hyum | 37m |
Connector code is already merged into Flink 1.14 and implements Flink’s new Source API with split enumerator and split reader to map Pulsar topic partitions to… | ||
| Advancing Data Mesh: Building A Stream Processing EcoSystem of Reusable Processors and Datasets | Andrew Nguonly, Wei Liu, Kasturi Chatterjee | 23m |
Netflix builds a reusable stream platform that runs in production and is GA as of Oct 2021, powering 160+ pipelines in production and using Flink + Kafka under… | ||
| Rundown of Flink's Checkpoints | david visikovic | 28m |
Explains how Flink implements Chandy‑Lamport–style distributed snapshots, compares HashMapStateBackend vs RocksDB (memory speed vs disk scale), and shows that… | ||
| Backfill Flink Data Pipelines with Iceberg Connector | Xinran Waibel, Sundaram Ananthanarayanan | 30m |
Netflix builds an Iceberg source connector on Flink Flip 27 source API that reads Parquet-backed Iceberg tables while mirroring Kafka-like ordering via a… | ||
| Flink's Table & DataStream API: A Perfect Symbiosis | Timo Walther | 35m |
Shows concrete code and a working demo: Kafka sources with three customer rows and duplicate transactions, a DataStream dedupe using keyed state plus timers… | ||
| Maintaining highly available Flink applications at scale | nagish fanali | 33m |
KDA implements a standby Flink cluster that is provisioned before stopping the live job, reusing the most recent checkpoints over savepoints and locking… | ||
| DrSquirrel - A self-service diagnosis tool for Flink jobs | Fanshu Jiang, Lu Niu | 22m |
DrSquirrel aggregates Flink metrics and logs into per-job snapshots (metrics reporter, Kafka log4j appender, a Flink job watcher that emits snapshots to Kafka… | ||
| Continuous Queries on Apache Pulsar using Flink SQL | Nan Lu | 25m |
StreamNative Cloud provisions an Apache Pulsar cluster and a Flink (Blink) SQL cluster, wires them together, and runs live SQL: a created topic feeds a SELECT… | ||
| Birds of a Feather Session - Stateful Functions | Seth Wiesman, Igel Shilman, Stephan Ewen | 40m |
Stateful Functions announces a shipping JavaScript/TypeScript SDK, enables a new async non-blocking transport by default, and adds user-defined metrics; remote… | ||
| nothing matches | ||