New Snowflake Capabilities and What They Mean for FoundationDB

on YouTube ↗11m

" summary

Snowflake adds table change stream and scheduled tasks that let an append-only staging table feed incremental merge logic into target tables while all task definitions, scheduling and queue state live in FoundationDB metadata/queue management. A demo shows streams reporting new rows, a task waking each minute to run a merge, the stream clearing and the target updated. Production load hits roughly 1.5 million task executions per day, each causing on the order of ten to a few dozen FoundationDB transactions.

AI-written summary · about

" chapters (4)

1. 0:00Intro
2. 0:21Snowflake Architecture
3. 1:35Snowflake Features
4. 4:24Demo

" related talks

• 22mHow FoundationDB powers SnowflakeDB's metadata!Ashish Motivala · FoundationDB Summit 2018
• 01h 20mSupercharge Your Real-Time Analytics: Snowpipe Streaming & Dynamic Tables in SnowflakeV. Prakash · DSC Europe 24
• 13mLightning Talk: Highlights of the Snowflake Elastic Data WarehouseTorsten Grabs · FoundationDB Summit 2018
• 23mBuilding A Code Generated Data PlatformDominic Colyer · DataEngBytes 2020
• 19mSnowcannon: Synchronous Data Replication in FDBMarkus Pilman, David Youngworth · FoundationDB Summit 2018
• 06mLightning Talk: Performance Testing of FoundationDB using TPC-CTrevor Clinkenbeard · FoundationDB Summit 2018
• 01h 30mSnowpipe Fundamentals: Ingesting Streaming Data into Snowflake Made EasyVed Prakash · DSC DACH 24
• 24mA Single Data Platform for All of Your WorkloadsStephen Weingartner · DataEngBytes 2021