AI and Edge Computing: Efficient, Lean, Dependency‑Free Language Processing
summary
The talk compares two approaches for doing language processing on-device without external services: self-hosting large open-weight generative models (targeting ~33–70B parameters), which preserves data locality but requires very large GPU purchase/rental costs and high energy use (example: ≈€100k GPU upfront, ~€20k/yr rental, ~3,000 kWh for office-hours), and a lightweight prompt-desensitization pipeline using local NER-based pseudonymization (Glenaer on DeBERTa ≈300M) plus a KNN compression classifier that runs on CPU/M2 Macs; the NER+KNN filter is fast and low-power but can miss high-level strategy leaks and needs hybrid rule-based blocks for sensitive intents.
AI-written summary · about
chapters (9)
- 0:00AI and edge computing intro
- 2:08LLM limits today
- 7:43Self-hosting models pros/cons
- 12:38Prompt privacy
- 22:07Non-neural text classification
- 23:30NER with small models
- 32:45Debiasing architecture
- 38:20Local inference vs service
- 46:01Conclusion
related talks
36mApplied AI on the edge
48mThe Power of Edge AI: Revolutionizing Future of Real-Time, Autonomous Processing
29mThe Go-to Language for AI: Opportunities and Challenges
41mEscaping the Cloud: High-Performance AI in your Browser
21mUnlocking the potential of low-power embedded devices with Edge AI software
22mBoring but Effective: WebAI That Ships
24mAccelerating AI on Edge
13mBuilding High-Performance AI Literature Processing