Serverless Osquery Backend and Big Data Exploration
summary
CloudFlare builds a serverless osquery backend called cenote on GCP using Cloud Functions, Pub/Sub, Redis, BigQuery, Stackdriver and memcache, decoupling ingestion from multiple subscribers. Real numbers: a log API test with 10,000 connections invokes ~250 functions and sustains ~100–150 active instances, a query run processed 43 GB in 1.4 seconds. Problems encountered include a Redis connection memory leak (fixed, reduced to ~50 MB) and RPC limits at >1,000 msgs/min; planned work includes file carving and Joe Sandbox integration.
AI-written summary · about
chapters (10)
- 0:00Introduction and CloudFlare overview
- 2:30Security challenges and OS query introduction
- 4:17Scaling OS query with SaltStack
- 8:00Data transport and storage with Cenote
- 10:52Serverless performance and constraints
- 13:40Handling errors and memory leaks
- 15:26Stackdriver for observability
- 16:36BigQuery for big data analysis
- 18:57Visualization and detection flows
- 19:18Future extensions and Q&A
related talks
24mScaling to Billions of Internet Observations per Day
32mUsing Serverless Functions for Real-time Observability
35mConvergence of Serverless APIs & Compute
38mServerless tools for machine learning, big data and app backends
38mBackground Processing, Serverless Style
41mUsing Serverless & ARM64 for Real-Time Observability
49mUsing serverless functions for real-time observability
44mHow to build a social network entirely on serverless