Building a new Big Data distribution based on Kubernetes with a twist!
summary
Claims an open source Big Data distribution that runs Kafka Spark NiFi HDFS on a Kubernetes control plane while replacing container runtimes with a Rust Kubelet using systemd and custom Operators in Rust. The stackable agent registers as a node, downloads Apache tarballs, writes systemd units, and streams journals back to the API. Demo shows 3-node Zookeeper and NiFi clusters deployed via k3s, the services start via systemd and the approach supports hybrid bare‑metal and container deployments with operators in Rust.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 4:01Problem with existing distributions
- 11:42Why Kubernetes and our approach
- 17:22Stackable agent and hybrid architecture
- 21:13Operators and operator framework
- 30:19Why Rust and our tech stack
- 34:05Live demo and deployment
- 40:26Q&A and wrap-up
related talks
35mModern Big Data Pipelines over Kubernetes [I]
38mRusty Container Ships: Kubernetes outside the Datacentre
24mDynamic Big Data in the Cloud with Kubernetes
29mKafka on Kubernetes: Does it really have to be the hard way?
33mKafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A]
48mrk8s: A Lightweight Rust-based Alternative to Kubernetes
05mIgnite: Is rust good for Kubernetes?
26mProduction Grade Kafka on K8s