Auto-sizing for Stream Processing Applications
summary
LinkedIn builds a controller called Sage that serializes prioritized sizing policies (memory, CPU, parallelism) and models interactions like remote-service latency and garbage-collection coupling. Sage runs as a stream app (~6000 lines of code), is enabled on hundreds of production jobs, and usually converges in 30 minutes to a right-size about 14% larger than hand-tuned optimum while preventing out-of-memory stalls and trimming the pervasive ~50% overprovisioning observed before.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:53Stream Processing at LinkedIn
- 1:26Problem
- 2:30Existing Solutions
- 3:29Apps use remote services
- 4:16Apps go beyond a DAG of operators
- 5:17Apps exhibit correlations in resource use
- 5:53Long-tail distributions of app characteristics
- 6:33Requirements
- 7:25Approach
- 8:52Sage Design
- 11:57Implementation Work in progress
- 12:41Conclusion Resource sizing is crucial for any service's performance, usability, operability
related talks
28mBeyond Prototyping: Building Production-Level Apps with Streamlit
11mStreaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving
28mStructure to Streams: Applying Full-Lifecycle API Management to ED Systems
26mWeek 1 Live Q&A
09mLightning Talk: How Silverflow Uses Backstage To Streamline Engineering and Compli...
35mStreamlining Development Processes through Self-Service Optimizations
38mInteractive Session: Flink on Autopilot with Splunk
44mMaking apps seem faster without optimizing