Fun with BIG Data with various databases
summary
Demonstrates concrete wins: importing 167 million rows into MySQL and changing one line to use partitioning cuts a query from ~25 minutes to 0.62 seconds; scanning raw SRTM tiles (1201×1201 16-bit files) creates shaded maps locally without a DB; combining MySQL fulltext filtering with XPath drops XML lookups to 0.4 seconds; recommends temp tables for bulk updates, avoid queries-in-loops, use views/indexes/caching, consider SQLite/Berkeley for flat datasets, S3/Solr/MongoDB and trying Akiban server to expose relational data over REST.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 1:13Who am I
- 4:53Relational Database
- 10:25Using stored functions
- 13:55Direct access to data
- 17:20Data analysis
- 20:35Simple program
- 22:13Getting bigger
- 23:28Managing lots of data
- 25:33XML files
- 28:34Tuning
- 29:10Changing data
- 31:28Selecting data from different tables
- 32:27Looking at the data differently
- 33:30Changing the numbers again
- 35:17XML
- 35:59XPATH
- 37:06Using XPATH
- 37:49Using XML
- 40:06Using Tables
- 40:36Postgres
- 40:55aquaponic
- 41:51error
- 42:48Acaba
related talks
23mFANTASTIC DATABASES AND WHERE TO FIND THEM
19mDatabases using R The latest
35mAn Ode to 17 Databases in 39 minutes
41mUsing Databases to Pull Your Application Weight
25mData Demons: Unleashing Chaos or Driving Performance?
41mAnalyzing Terabytes of Data with Google BigQuery
30mRetooling for a Smaller Data Era
44mConfessions of a DBA: worst and best things I've done in production