Introducing a Kubernetes Operator for Azure Databricks
summary
A new open-source Azure Databricks Kubernetes Operator lets Kubernetes manage Databricks notebook jobs as CRs: create a YAML that specifies notebook path, cluster spec (VM size and worker count), library deps and a secret mapping, then kubectl apply to run, list and stop jobs. Built in an eight‑week CSE engagement, it shifts from a Python sidecar to a single Go binary with a Go Databricks SDK, maps Databricks objects one‑to‑one, and requires Databricks host and token secrets.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 3:55Why build the Operator
- 7:05Operator architecture and iterations
- 9:56Setup and prerequisites
- 11:43Demo overview and notebook setup
- 13:05Configuring the Databricks job
- 14:14Running and monitoring the job
- 17:30Project status and next steps
related talks
41mDatabases on Kubernetes Using a Custom Operator: Day 1, Day 2, and Beyond
34mIntroduction to Kubernetes Operators for Databases
27mGetting Started with Yugabyte Kubernetes Operator
55mStop Caring about Kubernetes with Azure Container Apps & Dapr
46mDeliver apps from code to cloud with Azure Kubernetes Service
14mDemystifying k8s operators
50mThe hell of SQL DBs on Kubernetes versus the promise of operators
20mBuilding Internal Developer Platforms with Kubernetes Operators