MuEns: A Multimodal Human-Machine Music Ensemble for Live Concert Performance
summary
MuEns synchronizes a prerecorded piano part with live players by combining audio score following with camera-based detection of anticipatory body motion, then drives audio and visual output for ensemble cues. Timing control uses a stochastic spring-mass model whose time-varying stiffness parameters (gamma, beta) tune leader/follower balance; those parameters get adjusted during rehearsal. A backstage operator confidence monitor intervenes when score-following confidence drops. Deployed with a string quintet and Richter piano playback, concerts need brief predictable interventions, require highly skilled musicians, and still underrepresent expressive dynamics.
AI-written summary · about
chapters (13)
- 0:00Introduction to MuEns project
- 1:08Concert with string quintet demo
- 2:14Challenges in music ensemble systems
- 3:40Automatic accompaniment system limitations
- 4:42Visual and audio interaction in ensembles
- 5:44Leader-follower dynamics in music
- 7:09Adjustable coordination during rehearsals
- 8:13System implementation and visual cues
- 9:14Stochastic differential equation for roles
- 10:18Rehearsal adjustments and failure modes
- 11:44Audience feedback and system limitations
- 12:48Future work and musical nuance
- 13:10Conclusion and Q&A start
related talks
11mMachine Learning of Personal Gesture Variation in Music Conducting
41mCreating Intelligent Music Instruments with Machine Learning
08mNo merge conflicts
20mOpening Session 2
34mModelling emotional nuance of musical performance
01h 04mKeynote 3. Interactive generative models for music
05mInteractive Exploration-Exploitation Balancing for Generative Melody Composition
20mIntelligent Agents and Networked Buttons Improve Free-Improvised Ensemble Music-Making on ...