Huawei Cloud Stream Service in practice

on YouTube ↗38m

" summary

Beta cloud stream service uses Flink as execution engine and exposes a SQL client that compiles via Calcite, deploys containerized jobs through a REST API and collects yarn/flink/os metrics for costed visualization. Built from zero in about three months, it uses a distributed ingestion service (not Kafka), plans support for Kubernetes and Mesos and offers isolation and CI/CD. Notes limits: no smooth dynamic scaling today, millisecond IoT SLAs require edge coordination, and Calcite’s logical cost estimates miss runtime parallelism/memory tuning.

AI-written summary · about

" chapters (6)

1. 0:00Introduction and Huawei Cloud overview
2. 7:36Stream Service basics and architecture
3. 17:15Defining input, output and queries
4. 21:21Cloud-native design principles
5. 27:38Future directions and optimizations
6. 35:25Q&A and community engagement

" related talks

• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016
• 36mApache Flink in the Cloud-native EraStephan Ewen · Flink Forward Global 2021
• 20mGetting Started with Event Stream Processing via Apache Flink on Microsoft AzureIsrael Ekpo · ApacheCon@Home 2021
• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 43mTo Petascale and Beyond: Apache Flink in the CloudsGreg Hogan · Flink Forward Berlin 2016
• 26mRunning Apache Flink on AWSRyan Nienhuis · Flink Forward Europe 2019
• 23mCustomer Journey with streaming data on AWSRahul Pathak · Flink Forward Europe 2019