Exploiting Augmented Reality for Extrinsic Robot Calibration and Eye-based Human-Robot Collaboration
summary
A system uses QR-tagged virtual counterparts to compute extrinsic transforms between a HoloLens 2 with built-in eye tracker and a robot plus camera, enabling real-time gaze mapping and object segmentation; calibration runs 20 times per pose and yields most translation errors <5 mm and rotation medians ≈1.9°. Segmentation uses voxel-filtered Azure Kinect point clouds, K-d tree nearest gaze point and Euclidean clustering; evaluated IoU averages are 0.81 2D and mean 3D IoU 0.69, with recall 1.0 at IoU 0.5, and the demo successfully visualizes 3D bounding boxes in AR.
AI-written summary · about