Connecting the data infrastructure with the DataFlow
summary
Shows concrete integration: deploys Apache NiFi to consume MySQL binlog changes via Maxwell -> Kafka -> NiFi -> Elasticsearch, routes by JSONPath attributes, and indexes inserts/updates/deletes into ES in near real time. Explains processors (merge, compress, encrypt), backpressure and provenance guarantees, cluster behavior with ZooKeeper, and operational pitfalls (file-descriptor limits, TCP TIME_WAIT, no swap). Demo runs successfully and flow config lives in the flow.xml.gz for environment promotion.
AI-written summary · about
chapters (40)
- 0:00Intro
- 2:02Enterprise Integration
- 2:53Application Integration
- 3:42Application Coupling
- 4:26Integration Simplicity
- 5:02Data Format timeliness
- 5:45Communication
- 6:14Integration
- 8:40Integration Patterns
- 9:51What is an Efficient
- 10:14Configurable
- 11:12Secure
- 11:38Webbased
- 12:06Data provenance
- 12:42What is DataFlow
- 13:23DataFlow UI
- 14:25Flowfile
- 15:14Processor
- 16:25Connection
- 16:56Process Group
- 17:38Distributed Architecture
- 18:56Home knife
- 19:27Demo
- 20:55Maxwell
- 21:52Elastic Search
- 27:20New table
- 28:16Urbon attribute
- 29:32ETL
- 31:40Message
- 32:17Execution
- 32:53Compression
- 33:28Knife
- 34:31Time Wait
- 35:13Never Ever
- 35:46My Failure
- 36:23Questions
- 36:40Flow
- 37:08Flow XML
- 38:27Main use case
- 40:14How does it scale
related talks
46mApache NiFi 101: Introduction and Best Practices
47mBuilding Modern Data Streaming Apps with Open Source
44mNo More Silos: Integrating Databases and Apache Kafka
44mApache NIFi Deep Dive 300
43mA Data Streaming Architecture with Apache Flink #bbuzz
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data
17mDatabricks LakeFlow: A Unified, Intelligent Solution for Data Engineering
39mApache Iceberg ingestion with Apache NiFi