Designing Architectures: Big Data + Real Time
summary
A practical guide to designing and deploying a Big Data architecture that ingests and processes real-time streams using Hadoop. It compares open-source ingestion tools (NiFi, Flume, Kafka), storage and query options (Hadoop HDFS, Hive, Elasticsearch, Solr), and demonstrates an end-to-end pipeline by running the integrated stack against Twitter data.
AI-written summary · about
chapters (14)
- 0:00Start
- 0:52Who am I?
- 4:19Integrator, the toughest guy
- 6:15Big Data Ecosystem
- 7:58There is no monolithic solution, you have to play with the players
- 8:32And the truth is, we all mess up sometimes...
- 9:44Hadoop 101
- 11:38Classic Operational Infrastructure
- 14:18Main tools
- 16:29What if we want to integrate real-time events?
- 18:16Putting the puzzle together: selecting pieces
- 19:42Event ingestion: Flume
- 20:39Event ingestion: Kafka
- 22:28Event exploitation
related talks
49mBig Data Architectures and Ten Typical Pitfalls Along the Way
20mStreaming Architectures
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
47mBuilding Modern Data Streaming Apps with Open Source
53mBig Data: from mammoth to elephant
44mProcess and analyze millions of events per second with open source tools
30mArchitecting Applications With Multiple Open-Source Big Data Technologies