Apache NIFi Deep Dive 300
summary
Run ListHDFS on the cluster primary node and pair it with Fetch for content to avoid duplicate scans; for stronger ordering and exactly-once push records to an external durable queue like Pulsar instead of keeping all guarantees inside NiFi. Use the stateless NiFi engine for job-at-a-time idempotent processing, prefer record processors (Avro/schema/CSV readers) for speed and SQL-like transforms, use version control and the NiFi CLI/REST for automation, size repositories/backpressure, and refactor big flows into reusable process groups.
AI-written summary · about
chapters (11)
- 0:00Introduction to NiFi
- 1:33Achieving exactly-once data
- 5:08Stateless NiFi engine
- 7:18Integrating Apache projects
- 10:04DevOps and best practices
- 12:52Data records and optimization
- 15:54Diagnostics and monitoring
- 20:01Flow design antipatterns
- 26:08Advanced load balancing
- 29:07Refactoring and cleaning flows
- 37:20Q&A and closing
related talks
21mMeeting complex data load and data preparation challenges
42mConnecting the data infrastructure with the DataFlow
14mFast Data on-Ramp with Apache Pulsar on K8
39mApache Iceberg ingestion with Apache NiFi
42mApache Flink deep-dive #bbuzz
47mBuilding Modern Data Streaming Apps with Open Source
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data
36mHail Hydrate! From Stream to Lake