AI for element selection
summary
Demonstrates an appium plugin that uses a TensorFlow MobileNet-based classifier to find UI elements by semantics (cart, search, save), proving cross-platform Android and iOS clicks work end to end. The model runs about 0.8s latency per classification in the demo, can batch-classify thousands in ~0.2s in lab, recognizes ~68 labels today, and is open source models so teams can retrain on custom buttons. The result: fewer brittle XPATH/CSS selectors and reproducible clicks in real tests.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 0:45Mobile apps
- 1:19Google AI
- 2:00Appium had a brain
- 2:38What if
- 3:03Jonathan Lips
- 3:24Sauce Labs
- 4:31Demo
- 4:55Demo with Jonathan Lips
- 11:17Summary
- 11:50Speed
- 13:20How it works
- 13:42Machine Learning
- 14:09Open Source
- 14:56Multiple classifiers
- 17:25Who uses AI
- 19:00Chriss background
- 20:09Under the hood
- 20:21What is TensorFlow
- 21:17Pretrained networks
- 21:49MobileNet
- 22:18Tensorflow
- 22:54Optimal threshold
- 23:45Video
- 26:33caveats
- 30:12free shirts
- 31:31Selenium engineers
- 32:50Getting in touch
related talks
31mUsing ML to make your UI tests more robust
44mAI for every developer
18mHow AI Can Act as Top Layer of Appium Automation Runs
42mAppium: The Next Generation
30mHow AI is Transforming Test Automation
42mAI in React JS
42mAppium: The Next Generation
23mReimagining Mobile App Testing with Vision AI-Powered Agents