Microservice Pipeline Architecture
summary
A case study of BBC Newsbeat shows a content pipeline that turns fragile legacy backends into a resilient, prerendered system using Varnish S3 SQS SNS Dynamo CloudWatch and Ruby and autoscaling. It supports two million daily active users, enforces end-to-end invalidation within one minute, and reduces page assembly cost from ~30 HTTP requests to 3 by moving rendering into many small event-driven renderers that write HTML/JSON to S3 and a tiny composer that concatenates them. It trades cacheability for easier scaling and fast recovery.
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:02softwire
- 2:13BBC news beat
- 9:41REQUIREMENTS
- 9:59HANDLE HEAVY & SPIKEY TRAFFIC
- 10:58PUBLISH CONTENT QUICKLY
- 12:20CAREFUL WITH BACKEND SERVICES
- 14:26ARCHITECTURE TIME
- 16:28PROBLEM: HANDLING HEAVY TRAFFIC
- 17:33PROBLEM: CACHE MISS ON NEW & EVICTED
- 20:14PROBLEM: DATA LOAD FREQUENCY
- 23:07PROBLEM: UNNECESSARY RE-RENDERING
- 25:01PROBLEM: RE-RENDERING COMMON ELEMENTS
- 28:06PROBLEM: FRONT-END COUPLING
- 29:15COMPONENTS
- 31:06UNDER THE HOOD: RUBY AUTOSCALED EC2 S3 + DYNAMO
- 32:44PERFORMANT
- 33:26ROBUST
- 34:01ENCAPSULATED
- 35:33BEHAVIOUR BUILT INTO ARCHITECTURE
- 36:27EVERYTHING MUST BE CACHEABLE
- 36:57DATA MIGRATION
- 37:32CODE SHARING IS HARD
- 38:36MICROSERVICE PIPELINE ARCHITECTURE
related talks
39mMicroservices and Continuous Delivery at the BBC
24mMicroservices in Practice
31mUsing NodeJS to Build Micro Services Architecture
01h 06mA High-Performance Solution to Microservices UI Composition
56mFrom Homogeneous Monolith to Heterogeneous Microservices Architecture
29mNode.js Micro-Services
18mBuilding Netflix's New Video Pipeline
54mHow one team built their first microservice