Code and Technical Projects · 2021–present
MIME
A research platform that turns ordinary recordings of theatrical productions into a searchable corpus of performer poses.
- Role
- Principal Investigator
- Technologies
- Python, Computer vision, Machine learning, Pose detection
- Collaborators
- Peter Broadwell, Simon Wiles, Vijoy Abraham
- Organizations
- Stanford University Libraries
- Links
- MIME at Stanford
Machine Intelligence for Motion Exegesis
MIME analyzes recordings of theatrical productions frame by frame, mapping each actor’s body into a three-dimensional skeleton and recording the position of every limb, torso, and head. It works from ordinary video recordings, without motion-capture suits or specialized equipment. The result is a searchable database of body positions; the portfolio documents a corpus of 11 productions and approximately seven million body-position records.
The main interface presents each production as a timeline, showing how many actors are onstage, when the camera cuts, and where unusual movement occurs. A researcher can select an actor in any frame and retrieve similar body positions from elsewhere in the recording or across the full corpus. The system also clusters poses automatically and can search from either an uploaded image or a pose performed live in front of a webcam.
Research applications
In one study, the system analyzed 30 productions—ten each by Bill T. Jones, Romeo Castellucci, and Krzysztof Warlikowski—and attributed withheld productions to their directors using body-position data alone at a statistically significant rate. Another analysis identified where Delsarte poses appeared in contemporary productions. A study of six productions of Don Giovanni synchronized the stagings through musical pitch analysis and constructed a “consensus performance,” making it possible to measure when and how each staging diverged.
The project aims to make aspects of a director’s intuitive, embodied knowledge measurable. It can help historical researchers trace changes in staging conventions and help contemporary directors examine patterns and departures in their own work. This quantitative approach is intended to work alongside close reading and dramaturgical interpretation, not replace them.
Development
The broader research uses open-source computer vision, machine-learning, and pose-detection libraries to extract staging and characterization data from archival performances. It received a $75,000 seed grant from the Stanford Institute for Human-Centered AI in 2020 and a $20,000 Google Artists and Machine Intelligence research award in 2021. MIME was developed with Peter Broadwell, Simon Wiles, and Vijoy Abraham at Stanford University Libraries.
