Flink Forward Europe 2019
2019-10-07..2019-10-09
Berlin, Germany
| title | author | length |
|---|---|---|
| Run Interactive Queries with Apache Flink | Jiangjie Qin | 21m |
Flink adds an intermediate-result cache (FLIP-36) that persists blocking partitions to the shuffle service so repeated sub-DAGs avoid recomputation; a new… | ||
| The role Stream Processing plays for an Opti Channel Experience | Jamie Chittenden | 18m |
Bank implements a 5‑second SLA event pipeline using Apache Kafka, Apache Beam feeding Flink, HDFS for history and CDC via Informatica to expose personalized… | ||
| Dynamically Generated Flink Jobs at Scale | Regina Chan | 36m |
A metadata-driven Flink pipeline generates about 120K jobs/day to ingest, validate and milestone 90K datasets across a 700 TB lake handling ~7 trillion records… | ||
| Airbus makes more of the sky with Flink | Jesse Anderson, Hassene Ben Salem | 35m |
Airbus AirSense ingests ADS‑B position streams (about 12,000 aircraft at 2 Hz -> 24,000 msgs/sec, ~2 billion messages/day) and fuses satellite and ground feeds… | ||
| Query Pulsar Streams using Apache Flink | Sijie Guo | 47m |
Integration wires Apache Pulsar storage and schemas into Apache Flink so Flink SQL reads Pulsar topics as tables: namespaces map to databases, topics map to… | ||
| Real-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra | Ciesielczyk, Zontek | 21m |
Platform processes about 100,000 events per second at peak and makes data available at <1 second end-to-query latency. Pipeline uses Kafka → Apache Flink… | ||
| Not So Big – Flink as a true Application Framework | Oytun Tez | 37m |
MotaWord uses Flink as an application framework to manage translator workflows and real time invitations rather than as a batch stream processor it runs a… | ||
| Scotty: Efficient Window Aggregation with General Stream Slicing | Jonas Traub, Philipp Grulich | 24m |
Scotty implements general stream slicing to run window aggregation by adaptively choosing incremental strategies based on concrete workload properties… | ||
| From BaaB to EaaS in the Financial Industry | Piqueras Ruiz, Guerrero Sánchez | 37m |
Is it possible to kill the batch way in the Financial Industry?
Indizen technologies plays a key role in the architecture and innovation team of the first… | ||
| Bringing Cypher to Apache Flink | Sören Reichardt | 17m |
Implements a backend so Cypher queries run on Flink by translating Cypher to relational operators over an abstract table, shipping a working OKAPI backend that… | ||
| Real-time Experiment Analytics at Pinterest with Apache Flink | Parag Kesar, Ben Liu | 18m |
Real-time pipeline built on Apache Flink computes experiment metrics and runs statistical tests every 15 minutes, cutting metric latency from 24 hours to 15… | ||
| Do Flink on Web with FLOW | Dongwon Kim, Haemee Park | 22m |
A web service named FLOW lets users build Flink jobs on a drag-and-drop GUI and run them or download an auto-generated Maven project. Demo shows preview… | ||
| Unify Enterprise Data Processing System: Platform-level integration of Flink and Hive | Bowen Li | 40m |
Flink 1.9 ships a platform-level Hive integration that makes Hive metadata and data first-class: a new catalog API and HiveCatalog let Flink read/write… | ||
| Demystifying Flink Memory Allocation and tuning | Roshan Naik | 23m |
Ever tried to get get clarity on what kinds of memory there are and how to tune each of them ? If not, very likely your jobs are configured incorrectly. As we… | ||
| What's new for Flink's Table & SQL APIs? Planners, Python, DDL | Aljoscha Krettek, Timo Walther | 42m |
Flink 1.9 ships a pluggable Blink planner and a split table-module architecture so users can opt into Blink while falling back to the old planner; the unified… | ||
| A year in the Apache Flink Community | Robert Metzger | 16m |
Chinese usage edges out the US on site traffic and the Chinese community counts 13,300 DingTalk members plus meetups of 500+ and Flink Forward Beijing >1,000… | ||
| Kubernetes + Operator + PaaSTA = Flink @ Yelp | Antonio Verardi | 30m |
Yelp runs roughly ~50 clusters, ~1,000 jobs and migrates Flink off AWS EMR to Dockerized Flink clusters on Kubernetes, managed by a Kubernetes operator that… | ||
| Change data capture in production with Apache Flink | David Morin, Yann Pauly | 43m |
OVH builds a CDC pipeline that reads MySQL/Postgres binlogs with Go collectors into Kafka, processes and normalizes events with Apache Flink on YARN… | ||
| SQL Ask-Me-Anything | Timo Walther, Dawid Wysakowicz, Kurt Young, Bowen Li | 37m |
Unified Flink SQL now ships with the Blink planner in Flink 1.9 giving full TPC-H coverage and large batch speedups, with full TPC-DS targeted for 1.10. Hive… | ||
| What's new in 1.9.0 blink planner | Kurt Young | 37m |
Flink 1.9 adds a pluggable Blink planner that unifies batch and streaming under one API, swaps to a new binary row layout and type system, and implements… | ||
| FlinkDTW: time-series pattern search at scale using Dynamic Time Warping | Christophe Salperwyck | 42m |
An open‑source SIGKDD 2012 C DTW implementation is ported to Java and embedded in Flink to return TopK matches over past or streaming data, with pruning that… | ||
| Tips and Tricks for Developing Streaming and Table Connectors | Eron Wright | 25m |
Practical checklist for building Flink connectors: implement sources and sinks for the Streaming and Table APIs, use Flink’s… | ||
| AStream: Ad-hoc Shared Stream Processing | Jeyhun Karimov | 16m |
AStream is a distributed ad-hoc shared stream-processing layer built on Apache Flink that claims to reuse computation across thousands of short-lived queries… | ||
| Fighting phishing and spam with online machine learning on data streams | Wojtek Ptak | 23m |
FreshMail builds an online anti-abuse engine called SendGuard that processes thousands of messages per second and over 70M messages a day, using FlinkML and… | ||
| Flinking, Fast and Slow | Caito Scherr | 20m |
Built the company’s first Flink service in production that reads from Kafka, runs two Flink jobs (map/reduce then watermarking), writes to Kafka and an… | ||
| Streaming Processing Options in Google Cloud | Sergei Sokolenko | 20m |
Google Cloud offers a stack for streaming: Cloud Pub/Sub for global ingestion, Apache Beam/Dataflow as a serverless runner with exactly‑once semantics when… | ||
| Enabling Machine Learning with Apache Flink | Sherin Thomas | 28m |
In the world of ride sharing, many decisions such as matching a passenger to the nearest driver, need to be made in realtime. As a result, building and… | ||
| Towards Flink 2.0: Unified Batch & Stream Processing | Aljoscha Krettek | 40m |
Flink 1.9 delivers concrete steps toward a unified batch/stream stack: the Table API runs on the DataStream runtime via the Blink planner, sources move to a… | ||
| Batch/Stream Ask-Me-Anything | A. Krettek, T. Rohrmann, K. Young | 27m |
Speakers describe Apache Flink's effort to unify batch and stream processing by using a single engine and replacing the DataSet API with an extension of the… | ||
| Stream SQL with Flink @ Yelp | Enrico Canzonieri | 22m |
Yelp runs a service called Stream SQL built on the Flink SQL API that wires SQL queries into the existing Kafka + Avro + schema-registry pipeline, converting… | ||
| Building Self Service Platforms for Apache Flink | Canzonieri, Novak, Gorman, Nienhuis | 58m |
Panelists compare concrete platforms and tradeoffs: managed offerings (Amazon Kinesis Data Analytics, EMR), DIY on Kubernetes, and internal SDKs that wrap… | ||
| Making Sense of Streaming Sensor Data: How Uber Detects On-trip Car Crashes | N. Anderson, J. Yang | 23m |
Pipeline joins GPS, accelerometer and gyroscope streams via Kafka/Avro and Flink to run a TensorFlow RNN deployed through Michelangelo, processing under 10… | ||
| How to configure your streaming jobs like a pro | Gyula Fora, Matyas Orhidi | 41m |
Keep configuration minimal and practical: set max parallelism = highly composite numbers (120, 720) so key groups divide evenly; estimate parallelism from… | ||
| Flink for Everyone: Self-Service Data Analytics with StreamPipes | Philipp Zehnder, Patrick Wiener | 33m |
StreamPipes is an open-source self‑service pipeline builder that runs processors as Docker containers and deploys jobs to Apache Flink with Kafka as the… | ||
| Self-managed and automatically reconfigurable stream processing | Vasiliki Kalavri | 38m |
Presents two implemented systems: SnailTrail, an online critical‑path profiler that reconstructs execution graphs from instrumented events and computes a… | ||
| Multi-tenanted streams @Workday | Enrico Agnoli, Leire Fernandez | 36m |
Workday must never store or transmit two tenants’ data with the same encryption key while streaming large volumes (39+ million workers, hundreds of millions of… | ||
| Extending Flink state serialization for better performance and smaller checkpoint size | Grebennikov | 36m |
Shows that Kryo fallback often causes most CPU and state bloat, demonstrates registering custom serializers (protobuf, Scala sealed traits) and replacing… | ||
| Introducing Arc: A common int. lang. for unified batch and stream analytics. | Meldrum, Segeljakt | 18m |
Arc implements a common intermediate representation that cross-compiles pandas, Beam and other frontends into a single dataflow and enables optimizations… | ||
| State Unlocked | Seth Wiesman, Tzu-Li (Gordon) Tai | 33m |
Describes concrete state tools and migration guarantees: Avro schema evolution is supported (1.8) with reader/writer schema compatibility checks, Flink 1.9… | ||
| Running Apache Flink on AWS | Ryan Nienhuis | 26m |
Compares three concrete ways to run Apache Flink on AWS: Amazon EMR for managed Hadoop-style clusters, Amazon EKS for full-control Kubernetes deployments, and… | ||
| Introducing WinBro for Scalable Streaming Graph Partitioning | Zainab Abbas, Adrian Ackva | 18m |
WinBro implements the first Flink-native, scalable online graph partitioner that runs DBH and HDRF by combining windows and broadcasts to share degree state… | ||
| Deep Stream Dynamic Graph Analytics with Grapharis | Massimo Perini | 16m |
Grapharis implements a real-time deep graph learning pipeline using Flink + TensorFlow pipeline that trains and serves embeddings online without full… | ||
| Deploying Stateful FaaS on Streaming Dataflows | Adil Akhter, Marios Fragkoulis | 38m |
A system compiles stateful functions and their message flows into a streaming dataflow that runs on Apache Flink, making state local to operators and relying… | ||
| Event Streaming Architecture for Industry 4.0 | Abdelkrim Hadjidj, Jan Kunigk | 35m |
A blueprint for end-to-end Industry 4.0 streaming on Cloudera DataFlow connects edge agents to central analytics: Apache NiFi and MiNiFi collect and prefilter… | ||
| Apache Flink Worst Practices | Konstantin Knauf | 42m |
Practical checklist: choose DataStream for stateful apps, avoid Table API if you need stateful upgrades or late data handling, and pick delivery semantics… | ||
| Customer Journey with streaming data on AWS | Rahul Pathak | 23m |
AWS documents a concrete product timeline and usage: Kinesis Data Streams (2013) with three-AZ replication, Lambda (2014), Kinesis Data Firehose for… | ||
| Building an Apache Beam streaming runtime in GCP | Sergei Sokolenko, Reuven Lax | 46m |
Google describes building a Beam/Dataflow runtime that scales globally: it implements trailing-edge watermarks for correct low-latency triggers, a distributed… | ||
| Unlocking the next wave of applications with Stream Processing | Igal Shilman | 49m |
New SDK exposes addressable virtual functions that hold per-entity state, run on Flink, and serialize state with Protobuf for schema evolution. Messages route… | ||
| Build a Flink AI Ecosystem | Jiangjie Qin | 37m |
Claims a unified Flink-first ML stack that runs preprocessing, training and inference in one job by: a Table API-based ML pipeline (estimator/transformer API… | ||
| A Tale of Dual Sources: Pictures of Grief and The Job Manager’s Clock | Aaron Levin, Mike Mintz | 43m |
Stripe builds a custom Flink dual source that bootstraps from S3 archives and then hands over to Kafka, because Kafka retention is only 3 days retention while… | ||
| Streaming Event-Time Partitioning With Apache Flink and Apache Iceberg | Julia Bennett | 19m |
Netflix builds a Flink router that writes playback session summaries into Apache Iceberg tables partitioned by event time, using Iceberg atomic commits tied to… | ||
| Towards More Efficient and Adaptive Scheduling for Flink Batch | Bo Wang | 18m |
New Flink batch scheduler implements LazyFromSources plus an Adaptive Parallelism strategy that adjusts task parallelism at runtime by merging small partitions… | ||
| Intelligent Log Analysis and Real-time Anomaly Detection @ Salesforce | Andrew Torson | 29m |
Salesforce processes tens of gigabytes per second firehose and runs a public-cloud, elastic multi-tenant platform that performs real-time parsing, joins and… | ||
| Flink’s New Batch Architecture | Till Rohrmann, Ma Guowei | 44m |
Flink adds a region-based failover that restarts only affected pipeline regions (feature in Flink 1.9), plus a pluggable scheduler and a dynamic-slot model… | ||
| Stream Processing and Applications in the Modern Age | Stephan Ewen | 43m |
Flink reports a unified runtime push (notably Flink 1.9 with ~190 contributors and ~2,000 commits) and demonstrates a new project, Stateful Functions… | ||
| Open Source in the Age of Change | Chip Childers | 15m |
Open source must stop selling abstract ideals and start solving enterprise pain points: Cloud Foundry exists as a service project commercialized by IBM SAP… | ||
| Cloudera's Data-in-Motion vision | Marton Balassi, Andrew Psaltis | 22m |
Cloudera integrates Apache Flink into its hybrid cloud platform, committing to core streaming API support in December and a private beta launching next week… | ||
| One SQL to Rule Them All | Fabian Hueske | 40m |
Paper proposes a concrete, implementable model (accepted at SIGMOD 2019) that unifies batch and streaming SQL using three parts: time-varying relations as the… | ||
| From Zero to Production with Ververica Platform | Patrick Lucas | 16m |
A full demo shows a Kubernetes-native Ververica Platform install via Helm, creation of separate namespaces, and use of Minio S3 blob storage for jars… | ||
| Large Scale Real Time Ad Invalid Traffic Detection with Flink | Fares Sabbagh, Juan Gentile | 19m |
Criteo processes 300 billion bid requests and 4 billion displays daily and moves its invalid‑traffic rules from Kafka Streams to Flink, claiming faster rule… | ||
| Beam on Flink: How does it actually work? | Maximilian Michels | 38m |
Explains how the Flink Runner translates Beam pipelines: the portable runner sends a protobuf Runner API to a job server which stages artifacts and launches… | ||
| Build and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics | Hausmann | 30m |
Builds a runnable Apache Flink pipeline deployed to Kinesis Data Analytics for Java Applications, ingesting via Kinesis Data Streams and writing to Amazon… | ||
| Building a Self-Service Streaming Platform at Pinterest | Bairos Novak | 36m |
Pinterest standardizes a Flink-based self-service streaming platform running Flink on YARN that replaces ad-hoc Kafka Streams and Spark micro-batches, handling… | ||
| Writing an interactive streaming SQL engine and pre-parser using Flink | Kenny Gorman | 31m |
Claims that roughly 80% of streaming workloads map to SQL and shows a working interactive SQL editor plus backend engine called squealy o built on Apache Flink… | ||
| Running Flink in production: The good, the bad and the in-between | Lakshmi Rao | 36m |
The streaming platform team at Lyft has been running Flink jobs in production for more than a year now, powering critical use cases like improving pickup ETA… | ||
| When ordering matters | Niels Basjes | 36m |
Bol.com measures at the page-build point and routes events into Kafka partitions so downstream Flink can do stateful per-session processing with low latency… | ||
| Moving on from RocksDB to something FASTER | Matthew Brookes | 37m |
Compares RocksDB versus Microsoft Research’s FASTER as embedded state backends for streaming, benchmarking on EC2 SSDs with NEXMark queries and showing… | ||
| Flink Forward Europe 2019 highlights | 05m | |
Conference shows stateful functions announced alongside concrete upgrade paths: API 1.9 features and runtime work landing in 1.10. Airbus and other users… | ||
| nothing matches | ||