Text-to-Speech Data Preparation and Fine-tuning Workshop
summary
Workshop shows how to train Sesame's CSM-1B on a voice extracted from a YouTube video, using Whisper transcription to make ~41 clips of up to 30 seconds each, then fine-tuning in Colab with Unsloth + LoRA adapters (rank 32, ~2% trainable params). Training runs about one epoch (virtual batch 8), loss falls ~6.34 → 3.72. Result: zero-shot outputs are variable, cloning plus fine-tuning produces noticeably closer voice quality, and the fine-tuned adapters can be saved or pushed to the Hub.
AI-written summary · about
chapters (12)
- 0:00TTS fine-tuning intro
- 1:28Token-based TTS models
- 6:25Colab notebook setup
- 8:14YouTube audio extraction
- 10:42Transcription correction
- 13:51Fine-tuning with Unsloth
- 17:01Base model evaluation
- 22:02LoRA adapters setup
- 25:32Training process config
- 28:48Monitoring and saving model
- 30:54Fine-tuned model testing
- 33:05Workshop summary
related talks
35mTricks Learned from Scaling WhisperSpeech Models to 80k+ Hours of Speech
22mBuilding next-gen speech synthesis for Bookmate audiobook service
40mGathering Insights from Audio Data
01h 34mReinventing Speech-to-Text Transcriptions with Go and Whisper
25mBeyond Transcription: Building Voice AI That Understands Conversations
43mTalk by Om Thakkar (OpenAI)
53mSelf-Supervised Learning in Transformer Models for Birdsong
20mVoice fingerprinting & recognition