The Data Journey
summary
Tracks the move from siloed RDBMS systems to a shared data platform built on Lucene, Hadoop and an ecosystem that includes Spark and Kafka, claiming open-source adoption accelerates real-world uptake and scales to petabyte processing on cheap hardware. Describes building Nutch/Hadoop to replace impractical relational search, shows iterative MVPs beat waterfall silos, advises top-level buy-in, self-service data, starting with analytics and simple data science before machine learning, and says cloud/private cloud eases experimentation and production.
AI-written summary · about
chapters (11)
- 0:00Introduction to data journeys
- 2:14Evolution of software and data
- 5:07Open-source revolution and Lucene
- 8:41Hadoop and modern data tools
- 13:49Software and data in modern industries
- 15:41Challenges in adopting new technologies
- 17:31Cultural transformation for data adoption
- 20:23Practical steps for data journey
- 23:12Complexity of AI and data systems
- 26:05Role of cloud in data transformation
- 27:54Current state and future of data technology
related talks
53mBig Data: from mammoth to elephant
46mA Brief History of Big Data
34mTales of Data Architecture Evolution
02h 05mBig Data and the Data Lake from the Analyst Perspective
46mThe Evolution of the Open Data Ecosystem
16mHadoop's Impact on the Future of Data Management
45mModern Data Architecture
57mBig Data: Threat or Opportunity?