Emotion Estimation for Social Robotics
summary
System converts MediaPipe landmark output into 52 floating MediaPipe blendshapes, prunes them to 27 and feeds sequences into an LSTM 4-layer + FC classifier trained on the FER2013 dataset (augmented to 52k blendshape vectors). Model reports 71% accuracy and 62% AON score but collapses most labels into an “unknown” class, only reliably separating happy and sad. Experiments with Gemini and 1B transformers (via Hugging Face/JAX) are ongoing to improve generalization and multi-emotion classification.
AI-written summary · about
chapters (12)
- 0:00Emotion estimation intro
- 1:07Why estimate emotions
- 3:43Technical foundations
- 8:28AI emotion models
- 10:16Facial analysis with MediaPipe
- 13:12Dataset training
- 14:56Feature pruning
- 16:25Transition to large language models
- 17:50Experiments with LLMs and blend shapes
- 18:57Future improvements
- 20:06Live LSTM demo
- 20:51Summary
related talks
15mRecognizing Emotional Expression in Game Streams
32mDecoding Emotions from Art to Memes
01h 23mBeyond Sentiment Emotion Mining with Python and machine learning
11mAmbient Intelligence and Emotionally Intelligent Devices
39mThe art and science of making computers understand sentiment and emotion
49mAffective Computing – Bringing Humans and Machines Closer Through Emotions
34mEnhancing Customer Experiences with Emotional AI,
37mMultilingual sentiment analysis & emotion detection based on social media data