Data Rainbows - select * from cloud;
summary
Claims that Postgres Foreign Data Wrappers plus the open source steampipe let SQL query live cloud APIs with low latency and parallelism. Demo metrics: Zoom user list 159 ms, S3 listing 48 buckets then select star costs 2.6 s (hundreds of API calls), GitHub 3,500 commits in 7 s, IBM 2,500 daily quotes in 300 ms, GameStop hourly 1,900 rows in 240 ms. Built tooling supports paging, sub-API hydration, caching via materialized view to make repeated queries millisecond-fast.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:51Turbot in cloud governance
- 3:35Goal of querying cloud data
- 5:25Virtual tables and Data Rainbows
- 7:09Steampipe extends virtual tables
- 8:16Live cloud API queries
- 10:43Steampipe query execution
- 13:13API call multiplexing and paging
- 16:03Querying Twitter and GitHub
- 19:12Efficient large data sets
- 20:37Historical data and table designs
- 22:02Permissions and data access
- 23:47Materialized views tips
- 24:51Multi-cloud queries and joins
- 26:17When to use Data Rainbows
- 29:27Summary and Q&A
related talks
12mIntroducing Steampipe - select * from cloud
56mLightning Talks
31mMaking Data Dance
43mJust Use Postgres.
23mFANTASTIC DATABASES AND WHERE TO FIND THEM
41mUsing Databases to Pull Your Application Weight
34mAnalytics and AI with PostgreSQL - Mission Possible!
34mAnalytics and AI with PostgreSQL - Mission Possible!