DeepFisheye: Near-Surface Multi-Finger Tracking Technology Using Fisheye Camera
summary
A fisheye camera mounted beneath a touchscreen captures the whole hand and a deep network pipeline converts RGB to depth with Pix2Depth then a modified ResNet estimates 3D fingertip positions. Training uses 233k synthetic and 42k real fisheye images. The prototype reports ~20 mm average fingertip error with 80% under 30 mm, simple rule classifiers give ~83% contact finger accuracy and ~90% hand posture accuracy. Interaction demos run on tablet- and phone-sized target areas.
AI-written summary · about
chapters (10)
- 0:00Introduction and motivation
- 2:01Design criteria for NMFT
- 3:09DeepFisheye prototype overview
- 3:58Working principle explained
- 5:50Pix2Depth network benefits
- 7:25Dataset creation process
- 9:14Training and fine-tuning approach
- 9:36User study and evaluation
- 12:16Interaction scenarios demo
- 13:17Future applications and conclusion
related talks
07mFingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by ...
18mInfiniTouch: Finger-Aware Interaction on Fully Touch Sensitive Smartphones
11mEfficient Finger Model and Accurate Tracking for Hover-and-Touch, Mid-air and Microgesture Intera...
10mEllipses Ring Marker for High-speed Finger Tracking
16mInvestigating the feasibility of finger identification on capacitive touchscreens using deep ...
05mTouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images
20mToward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines
14mFinger-Aware Shortcuts