Building Apache NiFi 2.0 Python Processors
summary
NiFi 2.0 adds first-class Python processors that run with requires Python 3.10 and JDK 21 and wrap real libraries: Hugging Face, spaCy, PyTorch, IBM WatsonX SDK, psutil, faker, Pinecone/Chroma clients and LangChain. Examples include a compound GTFS reader that converts vehicle positions to JSON, a webVTT text extractor, an extracts company names processor, document chunking and vectorization, and Slack + Kafka outputs. Live demo shows parsing feeds, chunking to Chroma, querying vectors and posting results — all running end-to-end.
AI-written summary · about
chapters (24)
- 0:00intro
- 0:20preamble
- 0:34about tim
- 0:52flank stack weekly by tim spann
- 1:04future of data - nyc + nj + philly + virtual
- 1:26generative ai
- 3:05rapid innovation in the llm space
- 3:58cloudera generative ai stack
- 5:39apache nifi and real-time genai
- 7:18readyflow gallery
- 7:41cloudera + llms
- 8:28nifi 2.0.0 features
- 9:04python processors
- 12:00extract company names
- 13:47get compound gtfs data
- 14:50extract text from web vtt
- 15:37watsonx sdk to foundation
- 16:11system / process monitoring
- 16:58generate synthetic records w/ faker
- 17:41download a wiki page as html or wikiformat (text)
- 18:30get gtfs data
- 19:09other python processors
- 21:29demo
- 28:35thank you
related talks
59mPolyglot Pipelines with Apache Nifi
46mApache NiFi 101: Introduction and Best Practices
21mMeeting complex data load and data preparation challenges
44mApache NIFi Deep Dive 300
41mAdding Generative AI to Real Time Streaming Pipelines
30mThe Fabulous World of Data – ETL Powered by Apache NiFi and Scrapy
47mBuilding Modern Data Streaming Apps with Open Source
42mCloud Native Apache Pulsar Development 202 with Python