Drive-in Double-header: Datetimes and Log analysis
summary
Explains concrete rules and tools: store timestamps as use UTC for storage, prefer pytz and call pytz localize instead of passing tzinfo to datetime constructors, and avoid naive/aware mixing; shows parsing and analysis pipelines using IPython Notebook, matplotlib, CSV and pandas, plus Graphite for monitoring. Demonstrates parsing ~1.8M rows (filtering 200k in ~1s), disabling GC speeds parsing ~62%, computes percentiles and per-minute aggregates, and proves a metaclass change yields a clear linear speedup for large invoice-list requests.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:09Time is complicated
- 1:28Outline
- 2:12Time standards
- 5:31POSIX (UNIX) timestamp
- 7:21Two kinds of leaps
- 7:33Timezones
- 11:48calendar
- 13:04datetime - Do
- 17:14Interoperability
- 18:11PostgreSQL
- 19:43JavaScript
related talks
45mTimes and dates in Pandas
28mTimes and Dates in Pandas
26mHandling Timezones in Python
38mEventually, time will kill your data pipeline
40mPyData Stack: Pure Python open source data platforms
23mSolving Two Hard Problems in Computer Science Using Pandas
28mTick Tock: What the heck is time-series data?
32mScalable Python and SQL Data Engineering without Migraines