Build an e2e analytics application using DataHub
summary
A reproducible pipeline that makes Jupyter notebooks the deployable unit runs on OpenShift using Argo workflows executing papermill-run notebooks, S2I-built container images, and shared PVCs for file exchange. It ingests the Fedora mailing list, parses and analyzes text, writes JSON/CSV results to Ceph, and exposes them via Ceph S3-backed Hive external tables surfaced in Superset (word clouds, counts, engagement time series). Notebooks render artifacts and the daily/monthly DAG runs succeed end-to-end.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:24What is Open Data Hub?
- 1:37Building an ee analytics application using DataHub
- 4:56Application Overview
- 6:45Application Flow
- 8:29Defining UI - Ceph, Hive, Hue and Superset
- 9:50Dashboard and Data Exploration
- 11:56Automation Details
- 13:39Automating Jupyter Notebooks
related talks
28mDevOps for Data Science With Kubernetes
47mData Science in the Open Cloud Exchange model
34mData Products, From 0 to data science pipeline
24mUncovering Community and Project Insights Through Data Driven Methods
17mDemocratising data analytics
25mEdge-to-Core Machine Learning and Data Pipeline on k8s
24mOrchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow
30mContainer workflows for data science and ML