Roger B. Dannenberg

Our task is to hook a browser up to a microcontroller, preferably without running an entire web server on the microcontroller, which we want to keep simple.

Why would we do this? Adding interfaces to microcontrollers is hard. Typically, I modify program microcontroller code directly, upload the changes and test. It’s hard to tweak parameters, so usually I send raw data from the microcontroller to some nicer programming environment where I can interactively tune the behavior, such as input-to-output mappings, to my liking.

In this little project, I am going to create a little program to send MIDI data in response to accelerometer data. To “tune” the response, I will use a graphical interface in a browser to set parameters remotely in the microcontroller. To get MIDI out, I will use O2 messages. Browser-to-microcontroller and microcontroller-to-MIDI will be implemented using o2host, which already supports these tasks.

Here is what the interface looks like along with a short video demonstration of the entire system: microcontroller (ESP32 Thing from SparkFun), o2host, web page, and MIDI synthesizer:

Is this simpler than writing a special program for sensor data mapping? Well, that might require three programs: the microcontroller program for sensor data, a mapping program for mapping, and a browser program for the graphical interface. Maybe this approach is a little simpler.

The MIDI Controller Program

On the microcontroller, I'll use two degrees of freedom from an accelerometer. One axis, roll, will control pitch. The other, pitch, will enable or disable note production. We will parameterize the controller with the following:

• Roll Input Range - the minimum and maximum values for roll.
• Roll Output Range - the minimum and maximum output (MIDI) pitch.
• Pitch Threshold - crossing the threshold starts or stops note output.
• Scale Type - a selection of scales, e.g. Major or Blues.
• Blink - just for testing, sending “blink” will blink a light on the ESP32 controller.

About Accelerometers

Here, the “pitch” axis refers to a rotation dimension, not musical pitch: if you lean forward or lean back, you are rotating along the pitch axis.

Our sensors take advantage of constant gravitational acceleration. Gravity will produce acceleration along 3 different axes, which are sensed independently. The side-to-side acceleration is zero when the controller is level, but when you roll (slowly) to the left, the sensor detects a gravitational acceleration to the left, so we can report that as some amount of roll. Similarly, pitching forward applies some gravity in the front-to-back direction, so we have an independent sense of pitch. The third axis, yaw, corresponds to rotating the controller to face a different direction, which does not change anything sensed by the accelerometer. Therefore, we only have 2 axes to work with.

In principle, you can also integrate a gyroscope’s output to estimate yaw, but the math is complex and errors accumulate. Also, you can shake the controller, which will introduce acceleration in addition to gravity. This might produce some interesting effects, but we will make no plans for this and just see what happens.

Implementation

The entire project is open and free to inspect, including the o2host program, which is part of the O2 system, which is shared on Github. Sources for this project are here.

I was going to use Micropython, but I needed access to the accelerometer, and I already had code to do that in C++, so I did the whole ESP32 implementation in C++. I will focus on the “interesting” parts: communicating via O2lite and turning the accelerometer data into MIDI.

O2lite Communication

void setup()
{
  // Initilize hardware:
  Serial.begin(115200);
  Serial.println("calling setMinSecurity");
  WiFi.setMinSecurity(WIFI_AUTH_WEP);
  connect_to_wifi("esp", networkName, networkPswd);
  o2l_initialize("rbdapp");
  ...

This gets O2lite running and looking for a host, so you will have to run o2host (available from github).

To receive control information from the browser, we need to set up a “service” and some message handlers. The service is the top-level node of an O2 address. After creating the “midiapp” service, we create some handlers. I will show just the handler for limit which is a float parameter used in mapping from accelerometer data to pitch:

  ...
  o2l_set_services("midiapp");
  // Map from -axlimit to +axlimit to the pitch range
  o2l_method_new("/midiapp/limit", "f", true, set_limit, NULL);
  ...

...
float limit = 1.0;
...

...
void set_limit(o2l_msg_ptr msg, const char *types, void *data, void *info)
{
    limit = o2l_get_float();
    Serial.println("/midiapp/limit "); Serial.println(limit);
}
...

Creating MIDI

void loop()
{
    ...
    o2l_poll();
    imu_poll();
    double now = o2l_local_time();
    if (beat_time < now) {
        compute_beat();
        // advance to next beat AFTER current time
        while (beat_time < now) {
            beat_time += beat_dur;
        }
    }
}

Now for the fun part: mapping accelerometer data to make notes. Again, the algorithm here is very simple: First, we test if there was a previous note, and if so, we construct and send a note-off message for it. If pitch is zero, it means there is no note to turn off.

Then, we test the forward tilt in imu_ay (I will use “forward tilt” to avoid saying “pitch” which is confusing in this context) and if it is above thresh we do not want to make a new note, setting pitch = 0;. Otherwise, the roll data (in imu_ax) is clipped to plus-or-minus limit and interpolated to the range of min_pitch to max_pitch, and we call send_note to send it. We save pitch into the global prev_pitch so when we return to compute_beat we will know whether there is a sounding note to turn off:

// this is the "compositional algorithm" that constructs
// MIDI output from sensor data
void compute_beat()
{
    int pitch;
    if (prev_pitch > 0) {
        send_note(prev_pitch, 0);  // note off
    }
    if (imu_ay > thresh) {
        pitch = 0;  // disabled by roll
    } else {
        // use imu_ax for pitch and imu_ay for on/off
        if (imu_ax < -limit) imu_ax = -limit;
        else if (imu_ax > limit) imu_ax = limit;
        pitch = (max_pitch - min_pitch) * (limit - imu_ax) / (2 * limit) +
                min_pitch;
        send_note(pitch, 100);  // note on
        Serial.print("noteon ");
        Serial.print(pitch);
        Serial.print(" ");
        Serial.println(o2l_local_time());
    }
    prev_pitch = pitch;
}

void send_note(int pitch, int vel)
{
    // adjust the pitch until it is in the current scale
    int pc = pitch % 12;
    // scales are numbered 1 to 4, but index is 0 to 3:
    while (!scalemap[scale - 1][pc]) {
       pitch++;
       pc = pitch % 12;
    }
    // transpose out-of-range pitch if necessary:
    while (pitch > 127) pitch -= 12;;
    while (pitch < 0) pitch += 12;
    o2l_send_start("/midiout/midi", 0, "i", 1);  // tcp
    o2l_add_int32(NOTE_ON + (pitch << 8) + (vel << 16));
    o2l_send();
}

The Browser Interface

<table>
<tr><td><b>Roll Input Range</b></td>
    <td><input type="range" min="0" max="20" value="10" class="slider"
      id="rollrange" oninput="rollrange_input(this.value)"></td>
    <td><big id=rollrange_dec>1.0</big></td></tr>
...
</table>

This calls rollrange_input with a slider value from 0 to 20, and here is the function. Notice that we divide by 10 to scale the integer 0-20 scale to the desired 0-2 scale, and we write the numerical value into the last column of the table. Sending the value via O2 is a one-liner at the end (0 is a timestamp that means send immediately):

function rollrange_input(value) {
    var rollrange_dec = document.getElementById("rollrange_dec");
    value = value / 10.0;
    rollrange_dec.innerHTML = "" + value;
    o2ws_send_cmd("/midiapp/limit", 0, "f", value); 
}

To make this all work, we need to serve the web pages to the browser and forward O2 messages from the microcontroller to MIDI. This is done by running the o2host program and configuring it to forward the midiout service to MIDI (I chose the IAC Driver Bus 1 device). Also, we set HTTP Port to 8080; any value here will enable web services and O2-over-web-sockets. The root for web pages is www:

--------------------------------------------------------------------------
Configuration: acc2midi              Load   Delete 
    Rename to:                       Save   New 

Ensemble name:     rbdapp                            Polling rate: 500 
Debug flags:       d                                 Reference Clock: Y
Networking (up/down to select): local network      
HTTP Port: 8080  Root: www                            
MQTT Host:                                  MQTT Port:      
MIDI Out Service midiout              to IAC Driver Bus 1             (X )

··········································································
New forward O2 to OSC:          New forward OSC to O2:  
New MIDI In to O2:      New MIDI Out from O2:      MIDI Refresh:  
Type ESC to start, Control-H for Help.
--------------------------------------------------------------------------

Be sure to run o2host in the directory containing the www directory so that it can find and serve the web pages.

Conclusions

This post shows how you can use a web page to control parameters in a microcontroller, which normally has no convenient graphical interface. One thing I learned is that Wi-Fi is not a great low-latency way to play MIDI. I experience a lot of delays even with a dedicated router. O2 implements clock sync and in principle you could use “forward synchronous” timing to reduce jitter, but the recovery time for dropped TCP packets also seems very long, so fixing the problem with timestamps would add a lot of delay. Another option is BlueTooth, but my experience with BlueTooth is that it is hard to configure. It certainly does not have the kind of naming and discovery mechanisms of TCP/IP.

O2 supports an approach to building systems in a modular fashion, using multiple application programs with high functionality and controlling them through a flexible messaging system (O2). This illustrated here by splitting the application across a Browser for its graphical interface, a microcontroller because it is hand-held and has many sensing capabilities, and a synthesizer for making sound. O2host serves as a hub that ties it all together, allowing other components to use the lightweight o2lite protocol rather than a full O2 implementation.

Let me know if you need help with O2 or getting this example to work.

Additional references appear in the left sidebar (scroll up).