Messy Sensor Data: A Programmer’s Cleaning Guide
summary
Shows how to clean messy sensor weather data with runnable steps: use read_csv(sep, parse_dates, dayfirst, infer_datetime_format, na_values=['-']) to parse TSV logs, drop_duplicates(keep='last') then sort_values and set_index(datetime), convert compass directions to degrees via series.apply(dict.get), enforce regular timestamps with asfreq('30T') + interpolate(method='time') to drop or fill spurious 11/33-minute entries, plot with pandas/matplotlib, and record metadata flags for interpolated vs original values; notes caveats about time zones and merging different-frequency frames.
AI-written summary · about
chapters (10)
- 0:00Intro
- 2:00Motivation
- 3:31Reading CSV files
- 7:05Pandas way
- 12:43Reading the numbers
- 13:59Writing fancy functions
- 15:58Finding weird timestamps
- 18:10Plotting the data
- 20:01Interpolate
- 22:26Summary
related talks
23mSolving Two Hard Problems in Computer Science Using Pandas
01h 27mCleaning and Tidying Data in Pandas
01h 52mIntroduction to data analytics with pandas
01h 31mPandas, Data Wrangling & Data Science
42mMaking Sense Out of Flight Test Data with Python
34mPandas 101
28mWorking with dates and times in Pandas
02h 14mHands-on Python for Programmers