Automation Testing in a Real Big Data Project
summary
Describes an automation testing pipeline for a big-data advertising/RTB system: generate 10 million cookies, push them to S3, verify ETL into Redshift and Hadoop, and run realistic load through MongoDB sharded inserts, Redis bitmap queries and Aerospike reads (<1 ms). Uses Graphite/Grafana for before/after metrics; full historic validation can take 17 days but API/regression suites cut verification to minutes, proving data arrival, money flow, and external RTB integrations.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:59Big Data context
- 10:47REST API testing
- 15:25MongoDB sharding & tuning
- 20:05Redshift columnar DBs
- 23:37Redis real-time systems
- 27:16Metrics & fault tolerance
- 30:50Aerospike external systems
- 33:19Hadoop big data
- 34:42Testing recommendations
- 39:11Q&A
related talks
40mPerformance Testing and Big Data
33mAnalytics Data Driven Testing
30mHow to Test BigData/ETL Solutions?
23mChallenges of database testing
44mPerformance Testing with a "Big Data Tool"
25mTesting Data Pipelines
39mWhat Can Go Wrong With Testing an ETL Pipeline?
20mBuilding a Data Pipeline with Testing in Mind