Exploring alternative interactions in JavaScript

on YouTube ↗18m

" summary

Live demos show JavaScript hooking browser sensors to real models: webcam via getUserMedia into TensorFlow.js (PoseNet-powered Beat Saber clone that runs hard mode with wrist keypoints; hand-tracking Figma plugin; gaze-driven split-keyboard tied to a VSCode snippet plugin), audio spectrogram classification (water sounds triggering a 20s wash timer; claps vs speech), phone or Arduino motion as controllers over WebUSB/Web Serial, and consumer brain sensors detecting blinks to control a dino game.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 1:01Webcam data
3. 8:52Audio data
4. 13:31Hardware data
5. 17:05Outro

" related talks

• 42mExploring Alternative Interactions in JavaScriptCharlie Gerard · beyond tellerrand Berlin 2023
• 42mJavaScript in All the Creative PlacesStacey Mulcahy · Web Unleashed 2017
• 45mMind control in JavaScript - Experimenting with alternative interactionsCharlie Gerard · Cold Front 2018
• 30mLook mum, no hands ! Mind control in JavaScriptCharlie Gerard · DevFest Nantes 2018
• 28mBuilding Motion Controlled Websites in JavaScriptCharlie Gerard · HalfStack Online May 2020
• 41mFun With JavaScript and SensorsJan Jongboom · JSConf.Asia 2014
• 29mNatural User Interface using JavaScriptPrinciya Sequeira · JSConf Budapest 2016
• 24mThe next level apps for Web developers Machine learning on browsers 🤯Erick Wendel · BeJS Conf 2023