App-Based Task Shortcuts for Virtual Assistants
summary
SAVANT automatically maps natural language tasks to app UI screens by mining interaction traces (screenshots, render properties, gestures) and Play Store text, using keyword search plus intent parsing and a Dialogflow agent for slot filling. It launches activities when possible or replays traces to reach states, and is evaluated with the Rico dataset achieving 70.1% top-3 precision on 20 representative tasks. Limitations include verb/noun confusion, irrelevant domain matches, need for a system-level app, and open privacy and trace-sourcing challenges.
AI-written summary · about
related talks
26mOpen AI Assistants
27mApp Intents and the Future: If Siri Can’t Use Your App, Neither Can AI
44mSmartly AI: Cross Platform Voice Application Creation
52mDesigning digital assistants for those who need it most
12mScaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initi...
16mHow to voicify your Android app
15mSession 2: App Actions discovery
12mVOICE Summit 2019 | VED Talks | Dustin Coates | July 23