Decoding Emotions from Art to Memes
summary
Presents a multimodal pipeline that labels emotions in artworks and memes by extracting image features with EfficientNet and text features with DistilBERT, then fusing them via cross-modal multihead attention with dimension matching. Uses cosine-similarity ranking over a short emotion list, evaluated on ~1000 meme/image trials showing reasonable top-3 predictions but failures from cultural gaps and noisy captions; includes a local LLM postprocessor to generate crude natural-language descriptions.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:04Multimodal systems
- 3:56Multimodal data challenges
- 10:01Emotion detection motivation
- 11:03Data selection approach
- 14:34Pretrained feature extraction
- 16:45EfficientNet image processing
- 17:51Text tokenization features
- 20:02Cross-modal attention fusion
- 22:37Attention fusion details
- 25:11Final emotion integration
- 26:40Language models
- 29:13Meme challenges
- 31:00Conclusions
related talks
39mThe art and science of making computers understand sentiment and emotion
15mRecognizing Emotional Expression in Game Streams
21mEmotion Estimation for Social Robotics
01h 23mBeyond Sentiment Emotion Mining with Python and machine learning
39mAI + JavaScript Rocks
37mMultilingual sentiment analysis & emotion detection based on social media data
11mAmbient Intelligence and Emotionally Intelligent Devices
54mPresentation: Finding Your Meme Twin with Embeddings & Vector Databases