Using Data Orchestration for speed, reliability and real-time content
summary
Company builds Java layer that reads changes from multiple systems via RabbitMQ fan out and writes transformed JSON into Elasticsearch so websites query a small search API instead of every backend. Implementation uses a jobs component to subscribe, map fields and reindex (can be manually triggered), yielding near real time updates usually seconds to a few minutes and much faster page loads. Tradeoffs include data redundancy across providers and Elasticsearch indexing cost but the system is live and stable.
AI-written summary · about
chapters (9)
- 0:00Introduction and welcome
- 3:58About Randstad and challenges
- 16:46Data orchestration solution concept
- 18:32Why Elasticsearch for orchestration
- 22:28Jobs component and its role
- 25:35Search API and data access
- 28:03Live demo of the system
- 33:05Summary and benefits
- 35:14Q&A session
related talks
38mCrunching data in real time for Fortune500 clients
25mAccelerating Data Processing and Consumption for Building Modern Apps at the Edge
20mPractical Data Orchestration in Elixir
35mHighly Available Search at Shopify
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
14mJourney of Building a Scalable, Domain-Agnostic Order Management System
39mOrchestration of Big Data systems: A survivor's guide
01h 20mSupercharging Operations and Analytics: Using Spring XD to Support Analytics and CEP