Running PTP at scale
summary
Meta runs PTP to 10 microseconds precision across millions of servers and demonstrates a PTP stack handling 1.5M QPS to a modified PTP (ATSC) grandmaster and SPTP clients, with open hardware and FB clock APIs. Found failures: kernel watchdog scheduled as system work stops ticking (fix: do_work kernel thread with realtime priority), slow PHC reads reduced by caching high bits, and receive-path contention solved with seqlock for timecounter and standard SKB FIFOs.
AI-written summary · about
chapters (10)
- 0:00Introduction to PTP at scale
- 1:04Why PTP is needed at Meta
- 3:14Meta’s time synchronization hardware
- 5:09PTP stack and kernel side challenges
- 7:45Clock drift due to watchdog scheduling
- 10:17Reading PHC time efficiently
- 12:33Spin lock issues in time counter access
- 15:05FIFO implementations in TX timestamping
- 16:58Future testing and improvements
- 18:49Q&A on performance and latency
related talks
11mLightning Talk: Implementing the Precision Time Protocol (PTP) in the Zephyr Proj...
43mDo the Time Warp – the Rocky Horror PTP Show: Verification of Network Time...
13mUnderstanding Precision Time Protocol in Today's Wi-Fi Networks: A Measurement Study
40mTime-Sensitive Networking (TSN) with Mainline Embedded Linux
41mSetting up Real-time Ethernet and TSN With Linux: A 10-step Guide
17mntpd-rs: NTP for the modern era
16mRock around the Clock (Synchronization): Improve Performance...
55mPractical TSN; using NetChan for distributed PoC
