Developing for Big Data at the Intersection of Containerization and Infrastructure as Code

on YouTube ↗44m

" summary

Claims a repeatable path to let teams self‑provision Apache Hive on Hadoop by first building local containers then lifting to cloud, using docker compose for five‑container dev setups and terraform to provision eight EC2 instances (one name node, four data nodes, one Hive RDBMS, metastore, server). Includes a Python CSV cleanup, Beeline commands, a MapReduce wordcount smoke test that passes locally and on AWS, notes a 10‑minute transform and 3‑minute download, and shows teardown removes volumes and instances.

AI-written summary · about

" chapters (8)

1. 0:00Start
2. 3:17Richard Feynman
3. 23:40Put Together a Schema File
4. 30:48Smoke Test Script
5. 33:50Provisioning the Rdbms
6. 39:40Performance Graph
7. 40:16Takeaways
8. 42:12How Do You Learn

" related talks

• 35mScaling up to Big Data Devops for Data ScienceMarck Vaisman · PyData DC 2016
• 59mHadoop Essentials for the SQL Server ProfessionalBill Preachuk · 24 Hours of PASS: Summit Preview 2015
• 28mDevOps in a Data Warehouse: Inside OutAndy Burgin · DevOps Enterprise Summit San Francisco 2017
• 49mBuilding a 100‑Petabyte Analytics WarehouseAleksandr Mazurov · HighLoad++ Moscow 2018
• 07mComplementary Solutions with HadoopTasso Argyros · Strata Conference New York + Hadoop World 2013
• 46mDeploying and maintaining software for today's infrastructureMohamed Imran K R, Kiran Darisi, Srinivasan Rangarajan · Rootconf Cloud Server Management Miniconf Chennai 2017
• 45mSolving Real & Big (Data) Problems Using HadoopEva Andreasson · GOTO Aarhus 2013
• 44mAutomating a Big Data Platform with JenkinsSander Kieft · Jenkins User Conference U.S. East 2015