And now to something ELSE: Real Time Data Processing @ billiger.de
summary
billiger.de processes about 65 million offers and recomputes roughly six to eight million product updates per day using Python workers, Redis priority queues and an Elasticsearch product store. Claims: Redis queues use a tiny Lua extension to atomically claim batches of 1–200 items, workers scale horizontally (about 120 concurrent workers) and migration to Elasticsearch raises product lookup from ~12 to ~40–50 products/sec. The system deduplicates events, persists raw feeds separately, and migrates services without downtime.
AI-written summary · about
chapters (13)
- 0:00Start
- 1:46Key facts solute GmbH
- 2:54Behind the scenes of solute
- 3:42Price comparison: products & offers
- 5:53Workers event based processing
- 7:04Workers-event based processing
- 7:54The tank-event flow
- 10:22Redis priority queues
- 15:09Why migrate? Sharded Redis as data store
- 16:54Elasticsearch as data store
- 18:14Data Access Objects
- 20:00Migration from Redis to elasticsearch
- 21:23Conclusion
related talks
21mUse Python to process 12mil events per minute and still keep it simple (Talk)
34mHow We Handle Over 1 Trillion HTTP Requests ...
22mIt’s all about the db - a journey to accommodate 100k events per second
37mReactive Architecture on AWS: from 0 to 1.6M events/minute
41mE-Commerce search at scale on Apache Lucene
01h 00mHyper Speed: When Big Data Blooms
50mMake API Calls Wicked Fast with Redis
01h 02mBreak Out of Redis