Meeting complex data load and data preparation challenges
summary
Shows a concrete pipeline built with Apache NiFi that extracts from mixed sources, runs XML-to-JSON transforms, enriches records via lookups, and pushes batches into local Solr and Elasticsearch. Demonstrates actual NiFi processors used (XMLReader, record processors, custom price-association processor), an edge-core pattern to remove a bottleneck, and embedding processors in a ZooKeeper-backed cluster for fault tolerance and scaling. Claims faster development and resource-efficient, high-throughput indexing of semi-structured and unstructured content.
AI-written summary · about
chapters (5)
- 0:00Introduction and speaker
- 2:04Data extraction challenges
- 7:01Data processing pipelines
- 13:24Data flow showcase
- 18:36Solutions and implementation
related talks
46mApache NiFi 101: Introduction and Best Practices
44mApache NIFi Deep Dive 300
47mBuilding Modern Data Streaming Apps with Open Source
59mPolyglot Pipelines with Apache Nifi
30mThe Fabulous World of Data – ETL Powered by Apache NiFi and Scrapy
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data
01h 35mDeep Dive into the Big Data Landscape
43mBuild a Modern, End-to-End, Big Data Scala Reference Application