Roger B. Dannenberg

The o2host program can act as a simple web server among other things. Browsers that load pages from o2host can open a Web Socket and exchange O2 messages. This allows web browsers to communicate with other O2 processes, even though browsers do not have direct general access to network sockets, nor can they run the O2 protocol directly over TCP/IP network protocols.

In this blog, I will describe a little application that uses Handtrack.js to track hands and gestures using a webcam. Handtrack runs in Javascript in a browser. We will use Handtrack to control sounds created with Soundcool, an easy-to-use, interactive, modular computer music system. To get from the browser to Soundcool, we will use o2host as an intermediary (see Figure 1). The data flow is as follows: When a gesture is recognized and it is time to send an action to Soundcool, code in the browser sends an O2 message over a Web Socket to o2host. From there, the message is converted to OSC (Open Sound Control) and forwarded to Soundcool.

Demonstration Video

Here is what the system looks like when everything is running. The video analysis frame rate is low in the video because I was also running a screen capture program.

Implementation

The entire project is open and free to inspect, including the o2host program, which is part of the O2 system, which is shared on Github. Sources for this project are here.

To run the Handtrack code, I used examples as a guide. Here's what the main analysis process looks like:

function runDetection() {
    model.detect(video).then(predictions => {
        send_predictions(predictions);
        video.style.height = 240;
        model.renderPredictions(predictions, canvas, context, video);
        if (isVideo) {
            requestAnimationFrame(runDetection);
        }
    });
}
  

When an analysis is complete, send_predictions() is called with a list of descriptors for every object identified, e.g. there could be a face and an pinch gesture in the image. The implementation of send_predictions() has a lot of logic to determine when to actually start a sound and to decide which sound to play. The following has that logic replaced by ... in order to highlight the use of O2. o2ws_send sends an O2 message over the web socket protocol. The address will be something like "/sndcues/4/push2" which consists of an O2 service ("sndcues") followed by a complete OSC address ("/4/push2", used as the OSC address when we forward sndcues to an OSC URL and port number.) The other parameters you see are a timestamp (0 means now), the type string ("f" for float), and the value (1.0). Normally, the touchscreen app that sends OSC to Soundcool sends a 1.0 when a control is pushed, and a 0.0 when the control is released, so in this code, I send 0.0 to the same address after a 250 ms delay.

function send_predictions(predictions) {
    for (p of predictions) {
        ...
        if ((p.class == PINCH || p.class == POINT) &&
            ...
            o2ws_send(address, 0, "f", 1.0);
            setTimeout(() => { o2ws_send(address, 0, "f", 0.0); }, 250);
            ...
        }
    }
}

To make this all work, we need to serve the web pages to the browser and forward O2 messages to OSC. This is done by running the o2host program and configuring it to forward the sndcues service to OSC port 8001, and the drumloop service to OSC port 8002, both at 127.0.0.1 (the same machine). The o2host configuration looks like this:

--------------------------------------------------------------------------------
Configuration: handtrack             Load   Delete 
    Rename to:                       Save   New 

Ensemble name:     rbdhttest                         Polling rate: 500 
Debug flags:       rsq                               Reference Clock: Y
Networking (up/down to select): local network      
HTTP Port: 8080  Root: www                            
MQTT Host:                                  MQTT Port:      
Fwd Service sndcues              to OSC IP 127.000.000.001 Port 8001  UDP (X )
Fwd Service drumloop             to OSC IP 127.000.000.001 Port 8002  UDP (X )

················································································
New forward O2 to OSC:          New forward OSC to O2:  
New MIDI In to O2:      New MIDI Out from O2:      MIDI Refresh:  
Type ESC to start, Control-H for Help.
--------------------------------------------------------------------------------

Be sure to run o2host in the directory containing the www directory so that it can find and serve the web pages.

Finally, you'll need to configure Soundcool. There is a scconfig.soundcool file in the sources zip file that you can load, although you will have to reload the sound files (also included) because the files are stored in scconfig.soundcool using complete paths that will be different on your machine.

Summary

This post shows how you can use the o2host program to deliver messages from browsers to any OSC-enabled program like Soundcool. O2 supports an approach to building systems in a modular fashion, using multiple application programs with high functionality and controlling them through a flexible messaging system (O2).

Let me know if you need help with O2 or getting this example to work.

Additional references appear in the left sidebar (scroll up).