Real Time Analytics with MongoDB
summary
Replaces an 8–10 hour daily MySQL batch with real‑time counting at ~1,200 events/sec by writing raw hits into MongoDB and using atomic $inc upserts to keep per‑ad, per‑publisher tallies for daily, monthly and total views. Uses replica sets then sharding with a compound shard key (customerId + adId) to avoid hot shards, schedules the balancer to avoid IO spikes, and avoids mapReduce limits (single‑threaded JS, 10k grouping, writes to primary). Logs hit volume ~43M/day (~2GB indexes/day); next step is a Node.js REST API.
AI-written summary · about
chapters (4)
- 0:00Start
- 0:23Realtime Analytics with MongoDB
- 23:31choose your shard key wisely
- 37:16Business requirements
related talks
22mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
11mBig Data Analytics: Immutability & Idempotence
28mBuilding an Analytics Engine Using MongoDB and Go
48mHyper Speed! When Big Data Blooms
24mGive meaning to 100 billion analytics events a day, analytics at Teads
44mNear-Real-Time Event Analytics in a High-Load Project
47mScaling: from 0 to 25 million users
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns