Building a Multi-modal AI Application in TypeScript using ModelFusion
summary
A TypeScript kids‑story app built with ModelFusion turns speech into a full audio story with images and multi‑voice narration, using Whisper for 1.5s transcription, GPT‑3.5‑instruct and GPT‑4 for outline, title and prompts, Stability for images (~15s, ~25s total to image), and 11Labs/Element for TTS. Architecture uses Fastify, server‑sent events, asset storage (filesystem/S3), and streaming partial JSON so audio starts in ~10–20s instead of minutes; voices are chosen by voice embedding nearest‑neighbor on a vector store. Code and tooling are open source.
AI-written summary · about
chapters (11)
- 0:00Introduction to Storyteller app
- 1:06Challenges in AI story generation
- 2:37Architecture and providers overview
- 5:06Client-server flow setup
- 6:31First AI call transcription
- 7:34Generating story outline
- 8:38Parallel title and image generation
- 10:58Streaming structured audio story
- 13:44Voice selection with embeddings
- 15:31Text-to-speech synthesis
- 17:18Progressive content delivery
related talks
01h 30mBuilding Your First Multimodal Gen AI App 🚀
33mTranslate tech talks using open-source models and cloud-native Wasm
21mBuilding a Voice-Enabled AI Assistant With Javascript
30mLook ma, no hands! Multimodal AI-Agents in the browser
01h 07mBuilding and Scaling an AI Agent Swarm of low latency real time voice bots
19mLLM-Enhanced Multimodal AI: Revolutionizing Audio/Video Interaction
29mBuild and scale multiple Voice application by using TypeScript
23mBuilding with OpenAI: What's Ahead