Implementing the Lambda Architecture with Spring XD

on YouTube ↗01h 12m

" summary

Claims a runnable Lambda Architecture built with Spring XD, GemFireXD and HDFS and MapReduce that targets the ACM DABS smart‑grid challenge (40 houses, ~4 billion events). Pipeline: MQTT/RabbitMQ ingestion, Spring XD streams for the speed layer (duplicate removal, time‑slicing, three analytics: house and plug short‑term forecasts plus outlier detection), Hadoop MapReduce jobs launched by Spring XD for batch recomputation, and GemFireXD serving real‑time and batch views. Prototype runs on a laptop; cluster sizing and some integrations (in‑memory state, Redis access) remain work in progress.

AI-written summary · about

" chapters (32)

1. 0:00Intro
2. 2:38What is Lambda Architecture?
3. 3:29Approach is new, ideas not so...
4. 4:42Why Lambda Architecture?
5. 5:30Desired Properties of such systems
6. 7:07How to support those properties?
7. 8:19The Master dataset
8. 10:28Master dataset requirements
9. 13:02Desired properties on the Serving layer
10. 13:50Serving layer database requirements
11. 14:36Batch and serving layers
12. 16:31Speed layer
13. 19:03Incremental computation
14. 19:43Eventual Accuracy
15. 21:25One observation...
16. 22:52Approximation & Randomisation
17. 24:28Synopses structures
18. 26:32Stream processing (real-time function)
19. 28:18Generalised one-at-a-time approach
20. 29:31Micro-batching
21. 32:29Principles of Lambda Architecture
22. 33:31Implementing the Lambda Architecture
23. 34:53ACM DEBS 2014 Challenge Application
24. 38:27Data model
25. 39:51Analytical models
26. 43:22How others did
27. 47:46Implementation details
28. 48:59Our approach - ACM DEBS 2014 system architecture
29. 52:57Data ingestion
30. 56:21Stream actions
31. 57:07Spring XD streams
32. 59:22Batch Layer

" related talks

• 01h 20mSupercharging Operations and Analytics: Using Spring XD to Support Analytics and CEPJoseph Paulchell · SpringOne2GX 2015
• 53mNoLambda: A new architecture combining streaming, ad hoc, machine...E. Chan · Philly ETE 2016
• 48mLambda Architecture with Apache SparkTaras Matyashovskyy · JavaDay Lviv 2016
• 25mAccelerating Data Processing and Consumption for Building Modern Apps at the EdgeMarilyn Basanta, Ed Byford · Kubernetes on Edge Day North America 2022
• 49mLambda Architecture for High Data Volume and Fast Business ResponseHugo Rozestratem · GDG Campinas Data Fest 2017
• 30mSparkless Patterns: The New Data Stack and Modern Data ArchitectureAlon Agmon · hayaData 2024
• 01h 14mHow to build Big Data Pipelines for Hadoop using OSSCostin Leau · SpringOne2GX 2012
• 45mModern Data ArchitectureAlexey Grishchenko · JavaDay Kyiv 2015