VigNET: An intelligent camera app that assists you...
summary
An end-to-end Visual Question Answering app for people with visual impairment that uses Vision Language Transformer (ViLT) fine-tuned on VQA v2 (120k+ images, 600k+ questions) and returns answers in a fraction of a second with speech‑to‑text and text‑to‑speech. Prototype uses Streamlit, production UI uses React and an API built with FastAPI, deployed examples and GCP deployment instructions are open‑sourced. Demo shows many correct answers (accessibility score quoted) but notable failures on ambiguous celebrity images.
AI-written summary · about
chapters (12)
- 0:00Start
- 0:40Introduction
- 1:39Outline
- 2:41Visual Question Answering (VQA)
- 3:24Dataset
- 4:50Vision Language Transformer (VILT)
- 8:09Vision Transformer (VIT)
- 12:05Advantages
- 13:31Streamlit
- 18:14API Service
- 19:30FastAPI - Swagger Docs
- 26:51Try it out!
related talks
21mVizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World
01h 08mMastering Visual AI with Vision-Language Models & Advanced Evaluation Techniques
46mArtificial intelligence to the rescue of accessibility
45mBe My Eyes - Leveraging Generative AI to help the blind and visually impaired people
13mVid2Coach: Transforming How-To Videos into Task Assistants
15mStreetViewAI: Making Street View Accessible Using Context-Aware Multimodal AI
48mAn App That Sees, Hears and Responds
51mVisual Quality Assurance Using Vision-Language Models