SPRUCE it up! Open Source GreenOps at scale
summary
SPRUCE is an Apache-licensed pipeline that actually enriches cloud usage reports (AWS, Azure) with added columns for energy, CO2 and water and writes enriched Parquet files ready for Athena or DuckDB queries; it runs on Apache Spark (local, Docker or EMR), uses the Bu Vista API plus Ember and WRI Aqueduct for carbon and water data, and includes AI-inference estimates from Ecolog. The project ships fast (13 releases in a year, v1.1), handles files from ~0.5 GB up to 600 GB, and targets actionable reductions (e.g. reclaiming ~20% waste, region choice 384 g CO2eq vs 35 g CO2eq).
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 5:27Key green computing concepts
- 12:12Cloud Carbon Footprint overview
- 14:16Introducing SPRUCE project
- 25:24SPRUCE demo and dashboard
- 28:12Next steps and community
- 31:06Top actions to reduce cloud impact
- 34:19Resources and closing
related talks
19mKickstarting a greenfield data project
45mLaying the Foundation on OSS Environmental Sustainability
25mA simple serverless data pipeline
43mDeploying Large Spark Models to production #bbuzz
38mUnleashing the serverless data pipelines
01h 20mSupercharging Operations and Analytics: Using Spring XD to Support Analytics and CEP
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
50mDesigning a scalable data platform based on Apache Spark/Iceberg