How to Build an Integration Platform using Open Source
summary
Claim: a full integration platform is built using only open source and runs at scale for ~3 million users, ~3,000 suppliers and ~1,500 customers. Core pieces are API6 gateway (standalone pods, custom JWT plugin, CICD-managed configs), Keycloak IDP (multi-tenant, Postgres pooling quirks), NATS JetStream replacing Pulsar for durable event streams, and Node-RED integration flows for JSON transformations. Outcomes include a working zero‑trust JWT flow, custom Python config pipeline, known limits on user‑based throttling, and a planned move from Docker to Kubernetes.
AI-written summary · about
chapters (10)
- 0:00Open source integration platforms
- 3:13Company challenges
- 8:28External APIs use cases
- 11:18API Gateway and identity provider
- 17:54Keycloak setup
- 20:01Event streaming with NATS JetStream
- 21:22Data integration with Node-RED
- 23:52Completing the puzzle
- 24:54Future plans and monitoring
- 26:21Final verdict
related talks
29mOpen Source Powers The Modern Data Stack
28mBuilding an open source security... (Fletcher Heisler, Marcelo Elizeche Landó)
37mManaging the API lifecycle with Open Source technologies
30mHow to Integrate Open-Source Software into Enterprise Architecture?
28mOpen Source at Scale Building Operation in a large company
30mIs Building an Observability Platform with Open Source the Right Choice?
32mMastering Enterprise Open Source for Cloud and Vendor Independence
34mDemystifying Platform Engineering with OSS