Exploring the hidden potential of sound data
summary
Claim: browsers can do acoustic activity recognition by turning microphone data into spectrogram images and running lightweight CNNs. Implementation uses the Web Audio API to capture audio, Teachable Machine / transfer learning and TensorFlow.js to train/export models that classify sounds (minimum demo only needs 8 samples but accuracy improves with more). Prototype builds a live spectrogram, trains in seconds, recognizes brushing, coughing, snapping with some false positives; limits are model size, sample quality, concurrent sounds and privacy.
AI-written summary · about
chapters (20)
- 0:00Start
- 2:56Acoustic Activity Recognition
- 4:27What Is Acoustic Activity Recognition
- 5:41Microphones
- 9:08Visualizations
- 9:43The Frequency Bar Graph
- 10:23Spectrogram
- 11:07Visualizing Sound as a Spectrogram
- 22:13Sample Rate
- 24:41Transfer Learning
- 28:16The Transfer Learning Api
- 33:40Demo
- 43:03Interactive Storytelling
- 43:59Health Tracking
- 49:59Quality of the Samples
- 51:17Privacy
- 51:50Project Alias
- 53:40General Purpose Synthetic Sensors
- 57:09Collect Audio Data
- 1:06:50Do You Know any Application for Recognizing Birds by Their Song
related talks
20mExploring the hidden potential of sound data
32mAfter Censorship, Self-Censorship… but This Time It’s Funny, ...
28mHow to use Deep Learning to process audio data and classify sounds.
25mLooking at Sound
22mUbicoustics: Plug-and-Play Acoustic Activity Recognition
45mThe Sound of Your Secrets: Teaching Your Model to Spy So You Can Learn to Defend
19mSounds Aware: A Mobile App for Raising Awareness of Environmental Sound
38mIntegrating AI and Browser API Voice Recognition on Web App