Self-Supervised Learning in Transformer Models for Birdsong
summary
A custom bidirectional transformer trains on spectrograms with no tokenization, uses masked reconstruction as self‑supervision, and produces latent embeddings that cluster syllables into tidy loops; models use ~2.5M parameters and ~15 hours per bird of audio on consumer GPUs. Pretraining on ~500,000 recordings (Xeno Canto) generalizes to new species; a simple linear readout plus HDBSCAN turns weeks of hand‑labeling into minutes of inference and detects lesion‑induced stuttering. Machine labels predict future human labels better than humans predict themselves.
AI-written summary · about
chapters (10)
- 0:00Canary song research intro
- 3:43Canary song learning discovery
- 8:12Breeding season song reprogramming
- 12:33Manual syllable labeling challenges
- 16:10Self-supervised transformer for birdsong
- 21:52Latent space of syllable clusters
- 27:40Machine vs human labels validation
- 31:57Automated birdsong parsing
- 35:14Foundation model for vocalizations
- 37:24Future birdsong semantics
related talks
40mWho's singing? Automatic bird sound recognition with machine learning
44mScaling Semi-Supervised Production-Grade ASR on 200 Languages
28mHow to use Deep Learning to process audio data and classify sounds.
16mThe End of Awkward AI Transcriptions
01h 00mReducing the Dimension of Language: A Spectral Perspective on Transformers
26mRethinking the Transformer: Toward Native Multimodal Architectures
23mBirding with Python and Machine Learning
30mStarting deep learning projects without enough labelled data