Problems with Sensing Gestures

It’s been awhile since the last post. In between, I’ve been back to america and returned to Turin. While in the states, I worked with Mary on the sensor system, as I had a complete breakdown in the functioning of the gesture recognition system. It took us awhile to figure out what was going wrong, but eventually we figured out that it was two things: one a simple yet confusing data flow problem, the other a more conceptual issue with longer term ramifications.

The data flow issue took quite a bit of fumbling through to figure out. Mary was using the same patches and having no problems with getting models recognized in the software, yet I couldn’t get the system to interpret any data at all. But eventually the problem turned out to be an issue with Max/MSP screen order. This issue doesn’t usually raise it’s head with experienced Max programmers, but let this be a warning in complicated patches that these things can happen and really trip you up!

Now today I discovered thAt the training I did yesterday is a total bust because the sensor assembly has broken. The constant strain of use is beginning to take its toll on the equipment. Thus the musician/engineer/producer is running into maintenance issues. Another hat to don!

(speaking of hats, you might want to check out this video of the local army regiments called the “Bersiglieri,” who waer these amazing feathered hats and run while playing their band music. Really quite fun!)

The further conceptual issue (actually there are two of them, which I’ll get to…) has to do with what defines a gesture, and how we think of gesture in dance. When I showed Mary the gestures we we working with, she identified them as either consisting of mulitple gestures ( I guess you can call them “compound” gestures) or as being gestures that are not distinctive enough one from the other to be recognized as different. So now we think: okay, our human perspective on gesture is much more subtle and full of meaning than what a simple sensor can pick up. Sober probably have to attack this problem from multiple fronts. From a hardware perspective, what sensors or combination of them would work best for this idea? Certainly the couple that we are working with are inadequate for the huge range of movement that we are capable of. But then, of course, we run up against the problem of how to create a sensor system that is comfortable to move in., yet not too complicated to assimilate in terms of data processsing ( for real world use in the theater).

From a physical perspective, what do we perceive, internally, as a gesture? When does it start and end, what are the significant parts of it? What can be left out and still create meaning? And how much variation can there be before meaning is lost? So Mary was saying how the computer probably wouldn’t be able to distinguish among some of Renata’s gestures, yet why do they seem so different to us? Is it the quality of the movement? Is it just the vector picture of them? Are we able to accomodate to the sensing abilities of the sensors?

So to proceed from hre, practically, we have to identify those specific parts of the gestures we are working with, and train on those, in order to distinguish. But how much less limiting, and how much more meaningful, to do determinations of human meaning on the quality of the movement rather than a library of specific gestures. Such a library might be useful for every day communication, but not for an art form as sophisticated and subtle as contemporary dance. In writing this, I am struck by the need for some basic vector (shape) information, coupled with quality information, to build a complete picture.

So tomorrow, I am going back to the idea of training Laban movement qualities, to see if the computer can pick them up at all. Specific gestures are becoming somewhat useless, as te dance is changing midstream, as they are wont to do.

The other conceptual wrestling match we are engaged in is the Sound Spheres idea, and how to let it guide is and to represent it in our theatrical production. But this post is already long enough for one day… More tomorrow.

Day Two: “Shift”

I just found out today that the working title of the dance is “Shift.” Which is appropriate, as things have shifted a bit this evening. Today was a day of getting the sensors, and the machine-learning algorithm that they integrate with, set up and running on my own (I’ve always had my team of programmers with me, but now I’m here with only remote backup). But with a little SKYPE and email help from Mary, everything seems ‘golden’, as they say in Chicago.

As I wrote in yesterday’s post, his project is being sponsored by the VRMMP, and Paolo and Renata are producing this dance under some loose requirements that it contribute not only to Paolo’s professional development, but now it seems to also contribute to the development of VRMMP’s real-time modeling application called ‘MESH.’ I didn’t realize until today how important that part of the puzzle is, but after another wonderful Italian meal (featuring two special dishes, Burata, a super-rich cheese and cream ball served on a bed of pizza dough and arugala with tomatoes; and Zeppole, a wonderful cream-puff sort of doughnut dessert—I am SO full! Grazie, Vincenzo!!), a discussion with Vincenzo Lombardo, VRMMP’s director, revealed just how important integrating live performance into MESH is to this project. So, this is tomorrow’s challenge—to figure out OSC hooks into MESH and how real-time dance performance can integrate with this 3D modeling application. AND make it make sense with our SoundSpheres concept. An interesting ingredient for the artistic recipe…

Astral Technology moves to Italy

Renata Sheppard, Alumna of UIUC Dance, has been working at Turin, Italy’s Virtual Reality and MultiMedia Park (VRMMP), as part of a Fulbright fellowship. She arranged for me to come and work with them on a new dance, utilizing the sensors and gesture recognition system we developed as part of the Laban Movement Workshop we mounted with Thecla Schiphorst in preparation for the Astral Convertible Reimagined dance.

After a VERY long delay and flight to Turin, I met today with Renata and Paolo Armao, the sound designer at the  VRMMP, and began to plan our work. We started with goals that Paolo needs to accomplish, as VRMMP is sponsoring this dance and this is part of his release time from working on sound scores for films, which is mainly what they produce at VRMMP. I then explained in some detail about the sensors and the system I’m working with, and we began to bat around ideas for the concept of the dance.

At that time, Vincenzo Lombardo, director of VRMMP and a professor of computer visualization at the University in Turin, came in, and we discussed what we’d come up with. He also talked with Renata some about a talk she is giving tomorrow a the University, which it turns out I will also give part of, about my work on Astral Convertible and Laban. This came up last minute, but I’ll have plenty to talk about.

After this, we got back to work on our concept, which has a lot to do with David Sonnenschein’s concept of SoundSpheres. We’re thinking about a dance with a moving audience who experience the dance from four points of view: through three windows into the studio (which is on street level) and a fourth from a loft inside the studio. The sound score and dance will be inspired by, and illustrative of, the concentric circles of the SoundSpheres. The first window will be “I Am” and “I Think”, the second “I Touch” and “I See”, and the third “I Know” and “I Don’t Know.” The fourth will be a composite of all.

David Sonnenschein's SoundSpheres concept

Since these spheres are concentric, it makes some sense to have each of the succeeding windows influence the movement and sound of the preceding ones. We also are grouping them into twos, because the first two have to do with “internal, personal, private” sound, the second two with sound produced by physical action, and the third two with psychological and recognition aspects of sound. We’re thinking of having each window surmounted by the icon of the SoundSphere and its associated words highlighted, and handing out a pamphlet outlining the concept. Maybe we’ll have stamps for each viewer’s hand, too, so that when they collect all four stamps they will have seen the whole dance, which requires multiple points of view to experience in its entirety. Each window will have a set of headphones which will play the sound score associated with that point of view as well—these will be different from window to window, emphasizing the sonic aspects of each.

Paolo is going to be the “dub” master, taking the sound content that I will provide him, and mix and filter it according to his artistic tastes. This is how our sound collaboration will work. Renata is the choreographer, of course, and she will produce the video, which will also be related to the SoundSpheres, and interactive with the dancers as well. The Laban effort qualities will be the basis for the gesture recognition, and Mary Pietrowicz, who developed the gesture recognition system, will be on hand (virtually) to help out with gesture recognition programming. In this way, we are hoping to advance the application of the Laban system as a grammar for computer movement recognition.

“Logistical Carnival” Tuesday night

Last night we had our “Logistical Carnival,” as we called it, where we put the towers on stage with lights, speakers and sound system, Arduino boards with xBee wireless modems, and batteries. We also had projections on the cyc behind the towers as well as on the towers themselves. When the dancers arrived, we put a sensor on Annie F., to see how that was functioning. Then there was a semi-run through of the dance, to see the technical aspects of the production in relation to the movement in the choreography.

Before the dancers arrived, Chris were busy putting the lights together, as I was setting up a computer table for myself and Jihyuk Choi (xBee/wireless programmer) and Mary Petrowicz (machine learning from dancer-attached accelerometer data). Rebecca was helping with whatever she could, Ann DeVelder was present to help with accelerometers, Michael Williams was setting up projectors for Brett Jones and Raj Sodhi, Ken Beck was setting up the audio project boxes, Terri Ciofalo was organizing the scene, Lara Wilder was working on lighting, Donna Cox, Bob Patterson, and Alex Betts were experimenting with cyc graphics, Regina Garcia and Jamie were dealing with sets and helping out where they could, and I’m sure there were several other people, particularly in the stage management area, that were coming around to help set things up and get everything prepared for our tech experiments. It was a busy and very exciting and fun scene.

I was pleased with the technical experiments. The wireless communication worked in every place it was supposed to work for at least a significant amount of time. I was able to control the tower lights until at a certain time something happened with Max/MSP and my laptop and I lost control of serial data. I was also able to trigger tracks on the Daisy mp3 player, as well as turn on and off the sound from an mp3 player attached to another audio project box. So all of that was a success, at least in significant part. There is troubleshooting to be done, particularly as to why the wireless serial communication just stopped working after awhile, and why the Daisy’s sound output was so distorted. But I am confident we will get to the root of these problems.

The graphics for the production had there premiere tonight, and much discussion was had about them. The amount of ambient light coming from the towers is a bit problematic, as we’d though it might. But we got a good look at them in something of the eventual environment in which they’ll be seen. The light spill from the towers washed out the cyc quite a bit. So we’re going to try a black scrim in front of the cyc, and perhaps higher contrast graphical content.

The towers and tower projections were quite lovely, and Kathleen liked them quite a bit. So that, at least, is a keeper. There was minimal sound, and Kathleen and I were to have a meeting/listening session about sound on Tuesday (which we did).

There was much learned at the Logistical Carnival. Lighting, audio, and projections all produced interesting observations. And Mary was able to collect good dancer accelerometer data, as well as have  a little time to sneak in some time training her software. So it was a fruitful evening, indeed

Sensors and Circuits — What else?

On Monday, Mary and I went to the Astral Convertible rehearsal to begin the process of training with the newest version of her machine learning system. This involved first watching a run-through of the dance to further identify elements and gestures of the dance that we would like to train. Afterwards, we went to another dance studio with dancer Caitlin Lang, and attached two sensors to her, one to her right wrist, and one to her breast bone area. Mary wanted to try a sensor in the distall location, and one in a central location, thinking that the centrally placed sensor would be better at picking up on inversion and turning, and the distall sensor better at gestures and qualities or movement types such as swinging. We worked for about an hour gathering sensor data with Caitlin’s help. Mary then took the data and we will be working with it in the next few days.

Yesterday, Ken and I once again hit the decks with testing the audio circuits. We fired up the noisy DAISY mp3 player circuit again to see if we had actually solved the noise issue, and it seemed as though we had. But then we tried the same wiring with the identical second DAISY mp3 player circuit, and were back to the noise, as well as a bit of a distorted sound quality coming out the speaker. To make a long afternoon story short, we tried many different scenarios. As we were in the midst of these scenarios, the Arduino suddenly got very hot. So hot, in fact, that it melted the hot glue that was holding it onto the box in which it is mounted. So we unplugged everything, and after much testing, discovered that the xBee module had short circuited! Very strange, we thought, but true. When we switched out the xBee module, everything was fine. I don’t know if it happened when I issued two simultaneous serial commands, but I have a hard time believing that would cause a short circuit. In any case, all of this endless testing takes SO MUCH TIME! (Some day I’ll get to work on the music for this dance, instead of electronic circuits and systems… at least I was able to edit some samples I might be working with.) We still have some digital noise and transistor hiss in the sound, but for now we will have to live with it and move on to finalizing the last sound-producing circuit, the “Collins circuit” as we call it, in honor of Nick Collins of circuit-bending fame.

Tower Electronics

Yesterday Ken Beck and I and Jihyuk Choi worked together on completing construction and testing of two of the three technologies for making sound and controlling the lights on the towers. The first of these is the system that opens and closes a gate and controls input volume to the amplifier for an attached iPod. The central component of this system is an Arduino Duemillanove, with an xBee attached. The Arduino is commanded to open the gate and adjust the volume with Max/MSP software wirelessly through the xBee system. Jihyuk helped me to get the serial communication from Max/MSP to stream to the Arduino/xBee in its proper format.

After that system was functioning, we went to work on the system with the serial-controllable Daisy mp3 player. This system functions properly, with the serial message from Max/MSP choosing the tracks on the Daisy at a designated volume. Unfortunately, though, a LOT of electronic noise is coming through the speaker of this system.

One of the functions of the Arduino is to be the input for the photocells that are sensing light and shadows onstage caused by the movements of the dancers. The program we wrote has the Arduino send the value of the photocells every 1/5 second over the serial connection so that we can adjust the sensitivity values before each show. Jihyuk and I worked through the receiving and formatting of the serial message from the Arduino into Max/MSP (which was a bit of a difficult process, but thanks to an excellent tutorial in the program, worked out well). These serial messages were causing a lot of the digital noise, so when we turned them off (which we can do wirelessly), the system became quieter. But there is still a disconcerting amount of digital noise coming out of the speaker. We need to deal with this! Otherwise, the system is working somewhat slowly, but reliably.

Today, also, Rebecca Walter came in to help out with inventory of our many and miscellaneous electronic parts–very helpful!

Tomorrow, Mary and I will work on sensor placement and just how to begin training the machine learning algorithm for data input into the Max/MSP control system.

First post…long time coming

I and a wonderful team of collaborators at the University of Illinois Urbana-Champaign have been working for over a year now on this project, “reimagining” and restaging the dance work Astral Convertible, by choreographer Trisha Brown. I should have been documenting in a blog all along, but oh well, “better late than never.” Today’s post concerns yesterdays work, which was done at the wonderful Digital Performance Laboratory at NCSA. At noon, Mary Petrowicz and I met to view the video of the run-through of the dance, which was held in the Krannert Dance Rehearsal room on Friday Sept. 18. The aims of this session were to

1. Identity both qualities of movement, and specific gestures in the dance, that would lend themselves to the machine learning algorithms Mary has developed for use in the performance,
2. Identify group dynamics and behaviors that could also work in her machine learning system,
3. Determine the best location for a second sensor attached to the dancers’ bodies.

We identified a list, and next rehearsal we will hopefully get some dancers to wear the sensors and try to train the computer to recognize the qualities and gestures. In the meantime, Mary will be working on updating and extending her machine learning models to incorporate these new observations.

For a little interlude, Mary and I began explaining the sensors and the entire system design to Rebecca Walter, a graduate student at UIUC Dance, who will be assisting me with various aspects of the production.

Afterwards, I worked for several hours with Ken Beck, who is designing our electronic lighting control and sound producing/shaping circuits, to test and troubleshoot the new circuit boards he has assembled. The first one is a relay system for switching light circuits from the Arduino boards we are using as control boards in the set towers (www.arduino.cc). During this trial, I discovered that the serial code designed by our wireless systems programmer, Jihyuk Choi, is not transmittable by the limited capabilities of the serial object in Max/MSP, the control software I am using for the show. So I used Max to create the messages, and sent them to the Arduino wirelessly via CoolTerm terminal software instead. The serial code works perfectly, switching the relays to turn the lights on and off. So far so good.

The second circuit tested was the on/off volume control for the iPod connection to the amplifiers. This circuit also worked perfectly with the serial commands. The third circuit tested, a sound-producing circuit based on designs by Nicolas Collins, had some flaws, and Ken is going to trouble shoot later.

The work day ended at 7 pm. Much was done, much was learned, and there’s much to do.