Big Data and the Data Lake from the Analyst Perspective
summary
Claim: a data lake lets all data types coexist cheaply and be queried with familiar SQL tools. Demo uses the Hortonworks sandbox (Ambari, Hue), Hive/HCatalog for schema‑on‑read, Sqoop to import from SQL Server, and ORC files for speed; single‑node VM is slow but the same steps scale. Numbers: ~80% new data is non‑relational, Hive Stinger had 145 devs from 44 companies and 360k lines in 13 months. Outcomes: ORC + Tez makes queries interactive; Hive SQL on Hadoop, ORC optimized tables, YARN resource negotiation.
AI-written summary · about
chapters (13)
- 0:00Welcome to Big Data analytics
- 2:49Big Data and data lakes overview
- 12:00Traditional warehouses fall short
- 21:31Hadoop with SQL Server
- 30:22Hadoop adoption methods
- 41:18Hive and SQL on Hadoop
- 59:08Ambari and Hadoop services
- 1:11:38Lab setup and data ingestion
- 1:21:28Hive tables from raw data
- 1:33:29Excel via ODBC
- 1:41:24Sqoop SQL Server import
- 1:56:11Data exploration and analytics future
- 2:03:15Q&A
related talks
15mWhat is a Data Lake, Anyway?
22mBig data tools
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
58mUnderstanding Big Data for Software Engineers
57mBig Data: Threat or Opportunity?
34mData Lakehouse: Who, Why and How
33mThe advent of the open data lake
41mOn Data: Sky above, sand below, peace within