Users As Data
summary
Argues that user identities should be derived from users as immutable event streams rather than overwritten rows, so multiple user models can be folded over the same history; builds a lightweight data ecosystem (not a warehouse) using Esper + socket.io monitoring and a CLI aggregator on EC2, and runs batch analysis with Cascalog + R. Concrete win: a real-time monitor revealed about 1 in 10 requests failing after a TV spike, prompting fixes; network and flow diagrams from event data show where visitors go and which product areas underperform.
AI-written summary · about
chapters (12)
- 0:00Users as Data concept
- 3:57Clojure data philosophy
- 8:57Limitations of traditional data
- 13:37Data warehousing critique
- 17:54Computation vs data separation
- 20:21Users as interaction data
- 23:50Google's behavioral data
- 26:58Data timeline vs app lifecycle
- 29:06From warehousing to ecosystem
- 34:23Data ecosystem examples
- 40:21Summary: event-driven data
- 41:06Q&A: event stream centralization
related talks
38mPostmortem: Evolution of an Event recording solution
49mTransforming Analytical Data with Event-Driven Architecture
45mData as culture as software
49mData Modeling in an Event Centric World
29mIdentity Management in the Era of Big Data: The intersection of APIs, Hadoop & IdM
25mDealing with data in dynamic systems
11mBig Data Analytics: Immutability & Idempotence
40mOvercoming data infrastructure limitations with a new paradigm for ML