Forensic Log Analysis with BigQuery
summary
A workflow for hunting production incidents using logs, CSV extraction and Google BigQuery that finds needles in haystacks: convert Rails/nginx logs to CSV, upload to Cloud Storage, define schema and run SQL queries across millions of rows. Demonstrates handling half a million requests in under five minutes, pinpoints slow requests (examples show ~36,000 ms and 35,000 ms view renders) and fixes by adding pagination and an index. Uses BigQuery + Cloud Storage + CSV extraction and proves fast, cheap and shareable.
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:20Storytime
- 3:20What do you do
- 4:11Tools
- 5:21Too Much Data
- 6:18Forensic Log Analysis
- 6:49What is BigQuery
- 8:12Rain Off Demo
- 10:24How to
- 10:46Steps
- 11:10Reference
- 12:44Extract Interesting Bits
- 13:20Create CSV File
- 14:01regexes
- 14:32chomp
- 15:06csv
- 15:25Upload
- 16:27Schema
- 16:55Status
- 17:31Create Data Set
- 18:25Query Data Set
- 20:40Why Use BigQuery
- 23:06Thank You
related talks
40mFinOps Examples using Google Cloud BigQuery by Aliz.ai
31mExploring open data with BigQuery
55mCloud Analytics: from 0 to 100,000 events per second in 50'
25mData Demons: Unleashing Chaos or Driving Performance?
06mGoogle BigQuery for enterprise
24mGive meaning to 100 billion analytics events a day, analytics at Teads
46mSimple and Cheap Business Analytics with Google BigQuery
37mCloud warehouse monitoring ( Snowflake case study )