Paco Nathan (Liber 118) Interview - OSCON 2014
summary
Work with sensor arrays at scale — ADCs for aircraft bearings produce a claimed 12 exabytes a day for commercial flights, forcing real‑time analytics and edge/system tradeoffs. Industrial use favors probabilistic data structures and sketch algorithms (Twitter open source) to cut memory and bandwidth. Agriculture relies on in‑ground water sensors plus improving microsat satellite feeds (Planet Labs, Skybox) for actionable data. For processing, Apache Spark’s resilient distributed dataset (RDD) and streaming unify SQL, batch, graph, and ML so programs avoid manual partitioning.
AI-written summary · about
chapters (5)
- 0:00Industry focused data
- 0:46Large-scale sensor data
- 2:22Probabilistic algorithms
- 3:09Agriculture technology
- 6:33Apache Spark abstractions
related talks
19mA world of data
39mDatacubes: Enabling Space/Time Analytics and AI
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
33mOptimizing Speed and Scale of Real-Time Analytics Using Apache Pulsar and Apache P...
09mLightning Talk: Leveraging Large Scale Data Analytics on Cloud Foundry
44mProcess and analyze millions of events per second with open source tools
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
48mAccelerate science, research, and innovation with AWS