Bridging voter data and social data using Apache Streams
summary
Prototype system uses Apache Streams to normalize social and voter inputs, calls enrichment APIs (People Data Labs, Pipl) and processes results with Apache Spark before persisting caches in Postgres. Result: social handles become reliable unique keys, emails show higher match rates than phones or addresses, and some vendors yield surprisingly high true-match rates while startup priming costs are high. System proves feasibility for voter-to-social linking and surfaces caching, licensing, and graph-optimization as next engineering steps.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:06Roadmap
- 4:24What is voter data
- 7:02Issues with voter data
- 10:13Incremental approach
- 13:12Goals
- 14:46Motivations
- 18:36Challenges
- 21:39Data Engineering
- 24:00Apache Streams
- 25:36Highlevel tool stack
- 25:51Inputs
- 28:28Operation
- 29:33Integrations
- 30:49Limitations
- 34:00Wrapup
- 34:55Questions
related talks
29mEnhancing Vector Databases with Real-time Stream Processing
34mChasing the Stream Processing Utopia
52mSolving Analytical Problems using Apache Spark
32mBuilding a ChatGPT Data Pipeline with RisingWave Stream Processor and Cassandra Vector Search
01h 01mApache Spark Structured Stream: A Near-Real-Time Data Grinder
42mBuilding a near real time personalized feed using data streaming and Aerospike
01h 01mStream Data Processing for Fun and Profit
23mEnriching datasets using Apache Spark solves tracing in financial industry