Ingestify: Rethinking Ingestion for Complex Data
summary
Ingestify presents a concrete system: a Metadata-First Ingestion Layer that stores raw data as-is (S3 or local objects, Postgres/SQLAlchemy metastore) and emits Kafka events for downstream work. It uses last-modified checks to avoid refetches and detects no changes and skips fetch, so reruns ignored 35 tasks in the demo. Example numbers: an 8 GB tracking JSON downloads in ~3 minutes and single-core parsing takes ~4 minutes. The live demo runs and the pipeline actually produces immutable revisions.
AI-written summary · about
chapters (15)
- 0:00Welcome to Pyport
- 1:49Sports analytics challenges
- 3:34Centralizing data importance
- 5:26Row-based storage limits
- 7:16Traditional fetching inefficiencies
- 9:46Introducing Ingestify
- 11:37Ingestify configuration demo
- 13:01Football data with Ingestify
- 14:25Kafka integration
- 16:33Raw data reading
- 17:59Auto-ingest feature
- 19:05Streaming ingestion
- 21:44Open data with Ingestify
- 23:41Decoupled ingestion benefits
- 24:44Q&A
related talks
53mIngest node: (re)index and enrich documents in Elasticsearch
41mTreading Water in a Stream of Data
30mIngesting A Million Time Series Per Second On A Single Instance
32mBuilding a Real-time Ingestion Platform using Kafka and Aerospike
39mHow We Scaled Catalog Ingestion to Hundreds of Thousands of Entities
39mHow we isolate streaming ingest from search using RocksDB
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
01h 01mStream Data Processing for Fun and Profit