Lightning Talk: Enabling the Audio Modality for Language Models
summary
Hugging Face observes 2.7M models on the hub with ASR dominating and TTS rising since Feb 2025. Integration uses pretrained audio encoders (commonly Whisper large non-causal attention), which forces 30s limits, so streaming is built by chunking audio, summing chunk embeddings with token embeddings and using causal convolution cache or causal attention masks. Continuous audio maps to discrete outputs via audio tokenizers with codebooks (e.g., 32 codebooks, often using only 8); dependencies are handled by delay patterns or a second autoregressive transformer. Transformers library adds audio processors and a PreTrainedAudioTokenizer class for standardized reuse.
AI-written summary · about
chapters (12)
- 0:00Enabling audio for language models
- 0:21Audio in open source
- 1:27Audio in transformer models
- 2:51Audio vs vision models
- 3:32Audio sequence length challenges
- 3:52Handling audio input constraints
- 4:54Streaming architectures for audio
- 5:36Audio tokenization
- 6:59Audio in transformer library
- 8:01Standardizing audio-text models
- 9:26Audio feature extractors
- 10:30Summary
related talks
22mWhy TTS Models Now Look Like LLMs
27mWhat happens when AI starts grocking streaming audio directly
20mMultimodal language models
16mThe End of Awkward AI Transcriptions
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
58mMusic Generation via Masked Acoustic Token Modeling
36mNVIDIA NeMo: Toolkit for Conversational AI
40mKeynote: Open-source Multimodal AI in the Wild