Flink Forward Virtual 2020
| title | author | length |
|---|---|---|
| Keynote: Introducing Stateful Functions 2.0: Stream Processing meets Serverless Applications | S. Ewen | 41m |
Stateful Functions 2.0 separates compute from state so Flink keeps the state and messaging while user logic runs as remote functions (Kubernetes deployments or… | ||
| Keynote: The Easiest Way to Get Operational With Apache Flink | Konstantin Knauf | 24m |
Ververica Platform ships a free Ververica Platform Community Edition that runs wherever you have Kubernetes + object storage, provisions HA Flink runtimes via… | ||
| Keynote: The Evolution of Data Infrastructure at Splunk | Eric Sammer | 38m |
Splunk moves core processing from batch to an ingest stream by adding a firehose pub/sub and a Flink powered DSP that runs pipelines (SPL or Java SDK) for… | ||
| Keynote: Apache Flink - Completing Cloudera’s End to End Streaming Platform | Marton Balassi, Joseph Witt | 32m |
Cloudera integrates Apache Flink into a NiFi + Kafka pipeline and reports delivering most planned enterprise features: rewritten HBase and Kudu connectors with… | ||
| Keynote: Stream analytics made real with Pravega and Apache Flink | Srikanth Satya | 35m |
Combines Pravega, Apache Flink and Kubernetes into Dell EMC’s Streaming Data Platform and claims streams are a storage primitive that enable elastic segments… | ||
| State Unlocked | Seth Wiesman, Tzu-Li (Gordon) Tai | 32m |
Flink now guarantees evolvable state schema with Avro support since 1.8 and POJO migration since 1.9, letting you add fields (defaults become null) but… | ||
| Real time identity graph | Fakrudeen Ali Ahmed | 41m |
Adobe builds a real‑time identity graph on Flink that replaces a weekly 40 TB Spark batch job and runs at 50,000+ events/sec to serve 400M users across 2B+… | ||
| Production-Ready Flink and Hive Integration - what story you can tell now? | Bowen Li | 35m |
Flink 1.10 ships a production-ready Hive integration: a Hive catalog that persists Flink table metadata across HMS versions, reads and writes Hive tables views… | ||
| Needle in the Haystack - Monitoring health of a huge Kafka fleet with Flink | Chatterjee, Nienhuis | 15m |
Amazon MSK processes thousands of Kafka clusters’ JMX and host metrics in real time using Apache Flink running on Kinesis Data Analytics, keying by… | ||
| Managing Real time Data Teams | Jesse Anderson | 31m |
Moving from batch to real time is a complexity jump, not a lateral change: operations must support minutes-level SLAs instead of days, clusters like Kafka and… | ||
| Extend Flink to edge computing with much lower footprint | Youjun Yuan | 15m |
A lightweight Flink-compatible runtime called Quick framework compiles jobs into statically linked executables that run with runtime memory below 10 MB and no… | ||
| Testing Production Streaming Applications | Gyula Fora, Matyas Orhidi | 34m |
Shows how to test Flink from unit to full integration using Flink’s operator test harnesses for stateful UDFs, simple collection sources/sinks for batch-style… | ||
| Everything is connected: How watermarking, scaling, and exactly once impact one another in Pravega | Flavio Paiva Junqueira, Tom Kaitchuck | 42m |
Pravega implements a Flink source/sink connector that ties stream cuts and a reader group to Flink checkpoints so watermarks only advance when transactions… | ||
| Apache Flink in Production with Ververica Platform | Seth Wiesman | 43m |
Ververica Platform releases a free Community Edition that runs on Kubernetes 1.11+ with any S3/HDFS-compatible object store and exposes a REST API plus YAML… | ||
| Build your next-generation stream platform based on Apache Pulsar | Neng Lu | 28m |
Apache Pulsar provides a multi-tenant pub/sub with separated compute and storage, BookKeeper-backed layered storage, and geo-replication; claimed… | ||
| Flink SQL in 2020: Time to show off! | Fabian Hueske, Timo Walther | 46m |
Demo shows a Docker Compose Flink cluster (JobManager/TaskManagers), Hive metastore, Kafka, MySQL and S3 (MinIO) running live and succeeding: a Kafka→S3 CSV… | ||
| Building FeatureFlow, Criteo’s feature data generation platform | Piyush Narang | 20m |
FeatureFlow is a hosted Flink SQL platform that generates short‑term features for Criteo’s scale (~300 billion bids / ~4 billion displays per day) by running… | ||
| Apache Flink Worst Practices | Konstantin Knauf | 42m |
Concrete anti-patterns and concrete fixes: decide upfront whether you need at-least-once or exactly-once semantics because exactly-once requires replayable… | ||
| Flink acceptance testing and state compatibility checking | Catlyn Kong | 16m |
Yelp builds a docker-compose based Flink compose sandbox that spins up a standalone Flink cluster, Kafka, Schema Registry and test fixtures to run end-to-end… | ||
| Customizing exactly-once processing at scale with Flink | Andrew Torson | 27m |
Shows how Flink is extended to get exactly‑once at firehose scale by two concrete builds: a modified StreamingFileSink that enables Parquet row-group commits… | ||
| Data driven matchmaking streaming at HyperConnect | Jaehyeuk Oh, Gihoon Yeom | 23m |
HyperConnect rebuilds its 1:1 video matchmaking to run on Apache Flink, handling peaks of 5,000 req/s that translate to about 12 million candidate comparisons… | ||
| Geospatial Situation Detection through FlinkCEP @Uber | Teng (Niel) Hu | 30m |
Uber runs situation detection across 700+ cities using Apache Flink and FlinkCEP to match temporal sequences, with spatial clustering implemented as DBSCAN on… | ||
| High-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale | Jagannathrao Mudda, Ramayan Tiwari | 29m |
Netflix scales schema enforcement into Flink to generate and validate Avro records server-side for over 4 million events/sec and petabytes/day, using a… | ||
| How Streaming Helps Your Staging Environment and Sandboxes Stay Up To Date | Mike Solomon | 26m |
Shows a working pipeline that streams anonymized API requests/responses into Apache Flink and transforms them into an evolving mock server by retaining only a… | ||
| How Lyft built a streaming data platform with Flink on Kubernetes | Micah Wylde | 34m |
Lyft moves Flink from EC2 to Kubernetes and ships an open-source Flink Kubernetes operator that encodes deployment logic: it creates a fresh cluster, takes… | ||
| Stateful Functions: Polyglot Event-Driven Functions for Stateful Distributed Applications | Tzu-Li (Gordon) Tai | 55m |
Stateful Functions 2.0 on Apache Flink exposes an API and runtime that runs function instances as virtual keys, uses Flink’s consistent checkpointing to… | ||
| Implement Reliable, Isolated & Unified Job Submission | Zili Chen | 30m |
A platform running over Flink manages ~1,200 jobs that process ~35 trillion messages and ~4.5 PB/day and replaces flaky ad hoc submission with an atomic… | ||
| Flink's application at Didi | Xue Kang | 21m |
Built a Flink-based real-time platform and a SQL layer that lets non-experts write streaming jobs; StreamSQL based on Flink Table API runs alongside custom… | ||
| Building metric platform using Flink for massive scale at Netflix | Abhay Amin | 20m |
A config-driven Flink platform produces sessionized metrics at Netflix scale handling ~260 billion events/day and terabytes of RocksDB state for 150+ million… | ||
| Building Stateful Streaming Pipelines | Ankit Jhalaria | 19m |
Built a low-latency streaming data platform at godaddy using apache beam and flink on EKS that cuts data availability from 24–48 hours to near real time… | ||
| Lessons learned on Apache Flink application availability in a hosted Flink service | Hwanju Kim, Praveen Gattu | 35m |
Managed Flink apps run on EKS with RocksDB state backend and S3 for durable storage and availability is measured by refining Flink job state into a job… | ||
| Large-scale near-real-time (NRT) data analytics platform empowered by Apache Flink | Ying Xu, Kailash HD | 40m |
Lyft builds a Flink-powered ingest pipeline that pulls from Kinesis, aggregates into micro-batches, and writes columnar files to AWS S3 with multipart commits… | ||
| Distributed Processing for Machine Learning Production Pipelines | Robert Crowe, Ahmet Altay, REZA ROKNI | 38m |
Claims that a production ML pipeline is practical end-to-end: they implement TFX components (ExampleGen→StatisticsGen→Transform→Trainer→Evaluator→Pusher) on… | ||
| Machine learning with Flink in Weibo | Qian Yu | 18m |
Weibo uses Flink to unify batch and stream pipelines for recommendation: a single codebase replaces separate Storm/Spark/MapReduce paths, with online feature… | ||
| Cogynt: Flink without code | Samantha Chan | 20m |
Cogynt graphically composes CEP patterns and Cogynt translates them into a running Flink job that reads and writes Kafka topics via Kafka. Concrete rules… | ||
| Tuning Flink Filesystem Plugins for State Persistence | Scott Kidder | 20m |
Flink jobs with hundreds of stateful operators generate many small files that trigger GCS/S3 API rate limits and huge JVM churn because the Hadoop GCS… | ||
| Netflix Data Mesh: Composable Data Processing | Justin Cunningham | 22m |
Netflix extends Keystone into a real-time Data Mesh that moves petabytes a day and trillions of events across systems for 150M members, built on Apache Flink… | ||
| Integrate Flink with Kubernetes natively | Yang Wang | 37m |
Flink embeds a KubernetesResourceManager that calls the Kubernetes API to create and release TaskManager pods on demand, supports both per-job and session… | ||
| Equity and inclusion as a way of developing diverse, resilient, and more relevant OSS | Griselda Cuevas, Markos Sfikas | 23m |
Community survey plus project-metrics analysis and mentor friction logs reveal concrete gaps: 87% men in Apache contributors, only 3% women in wider open… | ||
| AI made easy with Flink + AIFlow | Jiangie (Becket) Qin | 38m |
AIFlow implements an event-driven workflow layer that turns logical AI nodes into executable jobs, chaining preprocessing training validation and serving into… | ||
| It’s finally here: Python on Flink & Flink on Zeppelin | Jincheng Sun, Jeff Zhang | 40m |
PyFlink exposes PyFlink 1.9 Table API and lets users pip install pyflink for Python 3.5–3.7, while native Python UDFs via Apache Beam portability arrive in… | ||
| Reliable, Real-Time AI with the Ververica Platform + a Kubernetes Operator in a Growing Startup | Austin Cawley | 20m |
FinTech Studios runs Flink on the Ververica Platform with a custom Kubernetes Operator that maps Ververica namespaces, deployments and savepoints to native K8s… | ||
| Rethinking of fault tolerance in Flink: what lies ahead? | Yuan Mei, Piotr Nowojski | 31m |
Flink adds unaligned checkpoints (code merged in the 1.10 branch, targeting 1.11) that let checkpoint barriers overtake in‑flight buffers so checkpoint… | ||
| Autoscaling Flink at Netflix | Timothy Farkas | 38m |
Autoscaling routers that consume a single Kafka topic are implemented as a separate Mantis job that polls Atlas every minute, uses statistical short-term… | ||
| Writing a Streaming SQL & Materialized View Engine | Kenny Gorman | 31m |
An interactive SQL engine built on Apache Flink and Apache Calcite parses statements with a SquealyIO parser, compiles POJOs into a Flink job, and samples… | ||
| Energize Multi-Tenancy Flink on K8s with YuniKorn | Weiwei Yang, Wilfred Spiegelenburg | 38m |
Runs Flink on Kubernetes with YuniKorn as a replacement (or secondary) scheduler, using Helm and an admission controller or labels to identify Flink… | ||
| Data Warehouse, Data Lakes, What's Next? | Xiaowei Jiang | 36m |
Alibaba uses Flink to run real-time ML pipelines and builds Hologres as a unified storage layer to avoid islands of HBase/Cassandra/ClickHouse. Hologres… | ||
| A deep dive into Flink SQL | Jark Wu, Kurt Young | 39m |
Explains how Flink SQL unifies batch and streaming via a pluggable blink planner that becomes default, translates Table API into a logical plan, uses Apache… | ||
| nothing matches | ||