Fun stories from building OpenRouter and where all this is going
summary
Company starts in early 2023, evolves from the Window AI Chrome extension into OpenRouter, an aggregator-turned-marketplace that offers a single API for 400 models across 60+ providers, processes 2 trillion tokens weekly, and recently raised $40m from a16z. It claims 10–100% month-over-month growth, cuts latency to 30 ms, standardizes tool calls and streaming cancellation, and ships an AI-native middleware plugin that gives every model web search and PDF parsing while improving uptime, routing, and granular model discovery.
AI-written summary · about
chapters (11)
- 0:00The Genesis of OpenRouter
- 1:16Initial Question.The story begins in early 2023 with the founder, Alex Atallah, pondering if the AI inference market would be dominated by a single player. He noticed the emergence of new models beyond OpenAI and a growing desire from developers to understand the nuances of different models, including their moderation policies [].
- 2:35The Rise of Open Source.The video highlights the beginning of the open-source AI race, with early models like Bloom 176B and OPT from Facebook []. A pivotal moment was the release of Meta's Llama 1 in February, which surprisingly outperformed GPT-3 on many benchmarks [], signaling a shift in the landscape.
- 4:38The Alpaca Moment.A major breakthrough occurred in March 2023 with the distillation of Alpaca. Stanford researchers demonstrated that by fine-tuning Llama 1 with outputs from GPT-3, they could transfer the style and knowledge of a larger model to a smaller one for less than $600. This proved that creating powerful, specialized models no longer required massive budgets [].
- 6:43Window AI.Before OpenRouter, Atallah launched Window AI, an open-source Chrome extension that empowered users to select their preferred LLM for any web application. This project laid the groundwork for what was to come.
- 7:18The Launch of OpenRouter.OpenRouter was co-founded with Lewis, the creator of the framework that Window AI was built on. Initially, it was a simple aggregator to collect models in one place.
- 7:57Growth and Evolution.OpenRouter quickly evolved into a marketplace, driven by the proliferation of model providers with varying prices, performance, and features. The platform has seen impressive growth, with a 10-100% month-over-month increase for two years. It now offers a single API for over 400 models from more than 60 providers [].
- 8:57Marketplace Dynamics.The transition to a marketplace was a response to the complexity of the growing AI ecosystem. By aggregating providers, OpenRouter helps developers achieve better uptime for both open-source and closed-source models and provides valuable data on latency and throughput [].
- 17:02Expanding Modalities.The future vision for OpenRouter includes incorporating models that can generate images and "transfusion models" that allow for conversations with images.
- 17:51Smarter Routing.The platform plans to implement more sophisticated routing mechanisms, including geographical routing and enterprise-level optimizations for GPU allocation.
- 18:07Enhanced Discovery.To help developers find the best models for their needs, OpenRouter aims to improve prompt observability, introduce more granular model categorization, and continue to offer competitive pricing.
related talks
18mAdvancing Search at Uber with OpenSearch
35mScaling RAG Applications to serve millions of users
24mThe Aryn Journey - Growing a Gen AI Startup on OpenSearch and the Road Ahead
14mReliable Agentic AI, the Open Way
10mGenesis Final Round: AI Square
10mThe Future for AI-First Enterprises is “Open” - How to Succeed and Not...
17mClosing Remarks 
20mBuilding an Open Core Business Model