From Bits to Tokens: Knowledge-Driven Generative Communication of Multimodal Data
summary
Proposes Knowledge-Driven Communication that sends tokenized semantics instead of bits, using multimodal foundation models and offline LLM-generated task operators to pick and tokenize task-critical regions, then conditions restoration on a receiver knowledge base and accumulated context. Implements a full prototype wrapping standard codecs (JPEG/H.264) for semantic pre-quantization, demonstrates 32 fps at 1080p on edge, achieves roughly 3x task performance at 10% packet loss, and matches accuracy with about half the transmitted bits.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 2:45Transmitting knowledge instead of raw data
- 4:54High-level architecture
- 7:46Knowledge base and restoration feedback
- 8:49Semantic pre-quantization
- 10:37Experimental results on images and videos
- 12:24Efficiency and latency optimizations
- 13:06Summary
related talks
20mMultimodal language models
58mMusic Generation via Masked Acoustic Token Modeling
10mLightning Talk: Not All Tokens Are Equal: Semantic KV-Cache for Agen...
21mMultimodal Open Source at Kyutai, From Online Demos to On-Device
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently
26mRethinking the Transformer: Toward Native Multimodal Architectures
40mKeynote: Open-source Multimodal AI in the Wild
55mDeep Emergent Communication: How It Started and How It’s Going