Using ML to make your UI tests more robust
summary
Shows a practical pipeline that uses Test.ai on top of Appium with a TensorFlow object‑detection model trained from Kaggle icon sets to locate UI elements and click them. Uses a 70/30 split for train/test, recommends ~1,000–5,000 icon examples per label, and targets ~90–95% detection accuracy. Demonstrations find search, settings and about buttons, run from CI, and require some regular Appium code and retraining; workable today but still brittle and not yet fully self‑healing.
AI-written summary · about
chapters (8)
- 0:00Introduction to AI testing
- 3:44Defining testing goals
- 7:58Training ML models for icons
- 13:40Overview of AI testing tools
- 18:55Setting up TestAI classifier
- 22:48Testing with generic icons
- 26:05Facebook’s Sapiens approach
- 28:30Current limitations and future
related talks
34mAI for element selection
43mSmart and Stable Automated Testing Using AI Based Dynamic Locators
30mHow AI is Transforming Test Automation
40mBeyond DOM: Leveraging Open-Source LLMs in an AI-Powered Appium Plugin
23mReimagining Mobile App Testing with Vision AI-Powered Agents
54mAI for Software Testing: The Ultimate Journey
35mApp Testing with AI Tools – Make it Easy
23mUI Testing for Fun and Profit