Government Big Data for All
summary
The talk describes the Panda Tobot project that identifies which public registers are published and provides access to ~250,000 users by ingesting massive government data dumps. It covers infrastructure decisions (managed Elasticsearch failing under load vs. self-hosted clusters with 32 GB JVM heaps and optimized EBS), real metrics (70 million judicial documents, stolen-car feeds updating every 30 minutes), and a pipeline of queues, microservices, change-detection and hooks/alerts for businesses and citizens.
AI-written summary · about
chapters (13)
- 0:00Open government data
- 3:09Open data legal status
- 5:21Pandatobot project
- 8:32Data selection quality
- 10:39Big data architecture
- 13:53Elasticsearch issues
- 16:08Data update speed
- 19:01Business automation
- 21:46Message queues optimization
- 24:48Data anonymization
- 26:38Business model outlook
- 28:23Government stance
- 30:52Open data use
related talks
10mBig Data and the Hypocrisy of Privacy
34mPast The Hype: A Better Way To Think About Big Data
40mThe Paradise Papers: Behind the Scenes
19mBig data in the service of reliable news
42mThe Texas Tribune: Making Data, and State Politics, Public
36mData-Driven Government and the Ruby Developer
33mData: The Land DevOps Forgot
18mData and the City