Python at Cloudkick
summary
Python runs most backend logic while metrics and time series land in Cassandra and relational state stays in MySQL; the monitoring agent is agent in C with embedded Lua that opens persistent TCP connections and pushes metrics. Architecture uses ~40 network services mostly in Twisted, RabbitMQ for AMQP routing with fanout/round-robin/tag policies, and Scribe for high-volume streaming to the live view. Deploy uses Fabric, CI with Buildbot, and a custom test runner gives ~50% speedup.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:47Fluffy features
- 4:57Cloudkick architecture
- 7:03Agent design
- 8:06Python as primary language
- 9:14Django web app
- 11:22Cassandra and Solar
- 13:11Profiling tips
- 14:36Logging with Sentry
- 14:57Twisted deep dive
- 18:29Twisted performance pitfalls
- 19:21Twisted error handling
- 20:06Metrics tracking
- 22:32AMQP communication
- 24:58Event framework policies
- 27:08Event-driven web flow
- 28:13Testing strategies
- 30:44Scribe for log aggregation
- 32:33Python in deployment
- 34:34Q&A
related talks
19mPython for Cloud Services and Infrastructure Management
33mLessons learned building Python microservices
43mTowards a full stack python monitoring+analysis framework
29mTunnel Snakes Rule! Bringing the many worlds of Python together to monitor Melbourne's biggest infra
39mArchitecture of a cloud hosting service using python technologies: django, ansible and celery
27mPython, Pyramid, and Embedded Devices
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
33mGive yourself time to sleep