Real Cross-Platform Workflow UI Automation
summary
Claims a working cross‑platform UI automation pipeline that uses a machine‑learning object-detection model on screenshots, sends inference to the cloud, and drives real OS input (smooth mouse and keyboard automation). It integrates web tools like Playwright/Appium/Robot Framework and visual tools such as Applitools, and it demos a full workflow: sign into GitHub, read a 2FA code from an Android authenticator, paste it back — browser + device across browser + Android phone and it works live.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:13What UI automation tests
- 6:30Native apps and IoT challenges
- 9:18Code selectors fail
- 11:50Visual selection tools
- 14:40Human behavior simulation
- 16:04Machine learning for UI
- 18:57Demo: Automating reCAPTCHA
- 23:20Demo: 2FA workflow
- 33:53Summary
related talks
33mReal Cross-Platform Workflow UI Automation
22mAutomating the difficult, real-world, multi-user scenarios
43mAutomating the real-user scenarios across multi-apps, & multi-devices
57mAppium: Automation for Apps
44mAI Driven UI Automation, API Automation and CICD Integration
30mAutomation: Simplifying for the future
31mScalable Windows Desktop Apps Test Automation - Myth or Reality?
44mAutomating Real-User Scenarios Across Multi-Apps, & Multi-Devices