The Gesture Follower allows for real-time comparison between a gesture performed live with a set of prerecorded examples. The implementation can be seen as an hybrid between DTW (Dynamic Time Warming) and HMM (Hidden Markov Models).
In most standard gesture recognition systems, gestures are considered as units that must be recognised once completed. Therefore, these systems output results at discrete time events, typically at the end of each gesture.
The Gesture Follower corresponds to a different interaction paradigm motivated by applications on expressive visuals and sound control: the system outputs “continuously” (i.e. on a fine temporal grain) parameters characterising the performed gesture.
Precisely, two types of information are continuously updated. These are probabilistic estimations of
1) the similarity of the performed gesture to prerecorded gestures (likelihood) and
2) the time progression of the performed gesture. The first type of information allows for the selection the likeliest gesture at any moment and the second type of information allow for the estimation of the current temporal index inside the gesture, referred here as “gesture following”.
These continuous output data are especially well suited for both selecting and synchronizing various continuous visual or sound processes to gestures.
OFFICIAL WEBSITE
http://ismm.ircam.fr/gesture-follower/
DOWNLOADS
The gf externals in MaxMSP is freely available as part of the MuBu library
REFERENCES
- F. Bevilacqua, N. Schnell, N. Rasamimanana, B. Zamborlin, and F. Guédy, “Online Gesture Analysis and Control of Audio Processing,” in Musical Robots and Interactive Multimodal Systems: Springer Tracts in Advanced Robotics Vol 74, J. Solis and K. C. Ng, Eds., Springer Verlag, 2011, pp. 127-142. [Download PDF]
- F. Bevilacqua, B. Zamborlin, A. Sypniewski, N. Schnell, F. Guédy, and N. Rasamimanana, “Continuous realtime gesture following and recognition,” in Gesture in Embodied Communication and Human-Computer Interaction: Lecture Notes in Computer Science (LNCS) volume 5934, Springer Verlag, 2010, pp. 73-84. [Download PDF]
- F. Bevilacqua, F. Guédy, E. Fléty, N. Leroy, and N. Schnell, “Wireless sensor interface and gesture-follower for music pedagogy,” in International Conference on New Interfaces for Musical Expression, New York, USA, 2007. [Download PDF]