Decoding Dolphin Speech
summary
A system adapts ImageBind-style multimodal models to force alignment of visual spatial tokens with audio by adding a region-matching loss and emergent dual attention heads that split sound vs language; the Dense AV variant produces localized activations for words and instruments. A pipeline uses ChatGPT to auto-annotate frames (about 432 million labels in 24 hours), fine-tunes Whisper with LoRA, and exposes results in a search UI called FinFinder. On dolphin data retrieval rises roughly 60× better than chance (6% 1000‑way, 23% top‑10); concrete recovered patterns include digging‑in‑sand echolocation buzzes and short side‑by‑side squeaking exchanges.
AI-written summary · about
chapters (7)
- 0:00Dolphin speech research
- 2:16Human words from video
- 15:24Dolphin communication techniques
- 27:32AI predictions for dolphin behavior
- 36:47FinFinder demo
- 55:42Future directions
- 1:05:42Q&A
related talks
57mDecoding Dolphin Communication
01h 01mIndividual Discrimination of Bottlenose Dolphin Signature Whistles Using Deep Learning
53mAI as a tool for discovery in animal communication
01h 04mGPT-Whale Language Models at Tools for Structure Discovery and Hypothesis Generation
19mLLM-Enhanced Multimodal AI: Revolutionizing Audio/Video Interaction
56mWhistle variability and social acoustic interactions in bottlenose dolphins
45mAuditory Intelligence : beyond speech recognition
01h 03mGenerative AI and What Is Meaningful Sperm Whale Communication