Exploring Alternative Interactions in JavaScript
summary
Prototypes show concrete inputs and code: webcam models use TensorFlow.js and face-api.js for facial labels, FaceMesh and pose for games (Beat Saber clone, Fruit Ninja) and a running-video controller; gaze input implements a gaze-based binary search keyboard and writes a React component via snippets in ~30 seconds; hardware uses web Bluetooth to read IMUs and EMG (Myo armband) and brain sensors (Emotiv, Neurosity/Crown) to control a Street Fighter clone; audio ML recognizes activities (handwashing counter) and cheek-tap gestures; all demos and code are open source.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 1:35What is HCI
- 2:16Webcam data in JS
- 4:23Facial expression demo
- 5:25Face landmarks for UI
- 6:53Eyebrow zoom demo
- 7:56Head typing with ML
- 9:21Full-body pose projects
- 10:43Beat Saber clone demo
- 11:45Simulator with pose detection
- 12:47Fruit Ninja clone demo
- 14:34Squid Game demo
- 16:31Figma with body movements
- 17:55Room-sized pose computing
- 18:57Gaze typing hands-free
- 20:22Gaze coding hands-free
- 22:25Hardware as input
- 23:09EMG muscle experiments
- 24:12Brain sensor interactions
- 25:59Brain-controlled Street Fighter
- 27:43Conductive thread UI
- 29:07Air drumming sensors
- 31:12Hoverboard motion demo
- 33:00Motion-controlled Street Fighter
- 35:47Audio data input
- 36:30Acoustic ML recognition
- 38:37Apple Watch prototype
- 39:40Facial touch sounds
- 40:44Closing thoughts and open source resources
related talks
18mExploring alternative interactions in JavaScript
45mMind control in JavaScript - Experimenting with alternative interactions
30mLook mum, no hands ! Mind control in JavaScript
21mHands-Free Coding With Gaze Control in JavaScript
28mBuilding Motion Controlled Websites in JavaScript
33mLook mum, no hands!
25mHands-free Coding with Gaze Control in JavaScript
42mJavaScript in All the Creative Places