Apache Liminal (Incubating)—Orchestrate the Machine Learning Pipeline
summary
Apache Liminal provides a declarative liminal.yaml that builds Docker images, registers pipelines as Airflow DAGs, and exposes simple HTTP endpoints via a python server image without changing user code. CI (Jenkins) uses liminal to push images to Docker Hub, submit Spark jobs to EMR, and deploy services to Kubernetes. Liminal emits metrics and registers CloudWatch alerts (job start, status, duration, missing runs). Demo trains a logistic regression on Iris, serves predictions and returns an 87% probability for a sample, showing the pipeline end-to-end.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:58Agenda
- 1:19Data Work
- 5:36Model Development
- 6:08MLOPS
- 7:04Data Engineering
- 11:28Challenges
- 12:10Integrated Platform Approach
- 12:53Integrated Platform Tools
- 16:14Diversity in Standards
- 18:36Liminal Technical Dive
- 25:43Liminal Example
- 29:01Python Example
- 33:41Liminal Pipeline
- 34:34Whats Next
related talks
33mInside Lila's AI Research Platform
36mDemocratization of ML Pipelines and Bringing ML Workflows to Heterogeneous...
42mCI/CD for Machine Learning
24mFrom Model to Production: Automating the ML Lifecycle with CI/CD
53mEnd-to-End ML
31mAIOps: Automating and Scaling ML and LLM Workloads in Production
05mCI/CD Pipelines and Machine Learning
30mBridging Classic ML Pipelines with the World of LLMs