Solving Two Hard Problems in Computer Science Using Pandas
summary
Shows how Pandas solves three practical headaches: restructure raw CSVs by splitting the concatenated device+parameter field into a normalized schema grouped by device type so each sensor yields fixed columns, handle extremely wide inputs (some PV sites had 8,000–10,000 columns) by per-device tables, and fix off-by-one billing errors for 15‑minute energy samples by aggregating with Period/PeriodIndex (use period ranges so a 23:45 reading belongs to the previous day). Includes Graphviz-based name analysis and a working refactor that made code smaller and correct.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:19Wind turbine data format
- 5:46PV plant data challenges
- 7:14Cache invalidation problem
- 10:47Naming conventions in data
- 15:44Off-by-one errors in timestamps
- 18:16Using periods in pandas
- 22:10Conclusion and Q&A
