Scalable Big Data stream processing with Storm and Groovy

on YouTube ↗01h 04m

" summary

Claim: a real-time pipeline handles social feed scale and delivers enriched tweets to a mobile app. Practical stack: Storm + Groovy + Kafka + MongoDB for ingestion, queuing, processing, storage; image extraction, deduplication and business-rule DSL run inside bolts; testing with Storm test API and Spock; caching with Guava for APIs; synthetic monitoring tweets every two minutes. Architecture details include spouts/bolts/topologies, parallelism hints, Trident exactly-once micro-batches versus Storm’s at-least-once, and handling bursts like 6,000 tweets/sec.

AI-written summary · about

" chapters (36)

1. 0:00Intro
2. 0:37PRESENTATION SUMMARY
3. 2:27HADOOP
4. 4:44Stream Processing Flow
5. 6:28STORM HISTORY
6. 7:09STORM ADOPTION Most mature, widely adopted framework
7. 8:00STORM PROPERTIES
8. 9:50STORM ABSTRACTIONS
9. 11:39TASKS Instance of Spout or Bolt
10. 12:13Stream Grouping How tuples are send between instances of spouts and bolts
11. 13:11Abstractions Summary
12. 13:35STORM DEPENDENCIES
13. 13:46DISTRIBUTED WORD COUNT TOPOLOGY
14. 14:43WORD COUNT Data Source
15. 15:09WRITING BOLT
16. 23:15STORM PARALLELISM storm main components
17. 23:47Parallelism configuration
18. 26:33Reliability API Tuple tree
19. 27:33Reliability API (Spout)
20. 29:55TESTING STORM
21. 30:15TESTING with SPOCK BDD style of testing
22. 31:40STORM COMPONENTS Physical View
23. 33:34STORM UI Monitoring and performance tuning
24. 34:47NIMBUS DOWN
25. 35:41SUPERVISOR DOWN
26. 35:53WORKER PROCESS DOWN
27. 40:03TRIDENT EXAMPLE Word count with Trident
28. 44:24STORM vs TRIDENT
29. 48:56CREATE TWITTER SPOUT
30. 49:47MEDPULSE TOPOLOGY
31. 51:44STORM ARCHITECTURAL BLUEPRINT
32. 53:19RULE ENGINE TOPOLOGY
33. 55:02Synthetic Monitoring
34. 55:59Use Graphite Statsd and Storm Metrics API
35. 56:34Performance Optimization
36. 57:42RESOURCES