Accelerating AI Application: NAVER's Journey from Model to Application
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2023 · on Wednesday, 2023-09-27· at 16:40· 3层 307会议室| 3F Room 307·
summary
NAVER productionizes models using BentoML to package framework-agnostic models with Python, deploy runners for GPU-optimized inference, and split API servers from heavy compute so CPU and GPU scale independently. The stack adds OpenLM for quantization, model parallelism, continuous batching and token streaming to raise throughput and cut latency. Reported impacts: unified model serving API, 3,000 community members, reduced deployment from eight weeks to three days, billions predictions per day.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:29ML Ops at NAVER
- 3:38Model serving challenges
- 7:15Off-the-shelf vs in-house
- 9:02BentoML design
- 12:18BentoML impact
- 13:22LINE’s ML expansion
- 14:50Neighbor’s BentoML
- 18:54BentoML deployment
- 23:01Production model serving
- 23:46LLM rise and OpenLM
- 25:58OpenLM features
- 27:27BentoCloud future
- 28:11Q&A
related talks
12mPreview: ML Serving
36mApplied AI on the edge
24mBuilding an ML Application Platform from the Ground Up
42mAI Everywhere– accelerate your development from PC to cloud
41mDeploying ML models in a Clojure environment
49mWriting GPU-Ready AI Models in Pure Java with Babylon
41mEscaping the Cloud: High-Performance AI in your Browser
16mAccelerating Web AI on