What structures facilitate programming interactive audio applications with ease? To help answer this question, I've created a set of programming problems. These are all small, but intended to illustrate some of the conceptual problems that arise within larger, more realistic applications, especially in interactive music compositions.
For each benchmark, I will write some code in
This is perhaps the most important benchmark. My experience says that "note" definition must be functional in style. The problem is to define a simple combination of audio generators and processors, in this case an oscillator, an envelope, and a multiplier, to produce a sound. The ideal case shows how elelements are combined by using a simple-to-read expression syntax.
Nyquist:
(defun note (pitch) (mult (osc pitch) (env)))
Ideal:
def note(hz):
return mult(osc(hz), env())
Boa:
Subsequent examples will show why note has to be an object rather than some other value.
class note (instrument): // instrument is superclass
def note(hz): // initializer method uses class name
var e
output(mult(osc(hz), e = env()))
connect e to self // capture "done" messages from env
return self
on durationd: // pass durations to envelope
e <- set "durationd" to durationd
on gatef: // pass gate values to envelope too
e <- set "gatef" to gatef
The following method could be inherited from instrument:
on donea: // when env is done, note is done
set "donea" to self // send out Aura message to whomever
// is connected to this object
Once a note is defined as an abstraction, you want to be able to create an instance of the abstraction and play it. One challenge here is to make this simple. For example, it would be bad if you had to create a note, patch it into some signal sink, start the note, stop the note, deallocate the instance, etc.
Nyquist:
(stretch dur (note C4))
Note that stretch changes the environment in which note is instantiated, thus duration is visible to note and all its parts.
Ideal:
mynote = note(C4) mynote <- set "durationf" to dur
The idea is we want to send a message to an Aura object represented by an Aura ID. The notation id <- set attribute to value is the way I will indicate sending an Aura "set" message to an Aura object.
("durationf" is not a typo. All attributes end with a one-letter type identifier because Aura attributes are statically typed. The explicit type letter harks back to Fortran, but it seems like a good approach from a human factors standpoint. This is an Aura issue, not something this document plans to address. The types are "a" Aura ID, "i" integer, "f" float, "s" string, and "b" boolean.)
Overall, this code won't work because note() returns an instance of mult(), which knows nothing about durations. (At least this is true in a simple implementation.) I do not know how to implement this ideal language, but it is still useful to think about what we would like to write.
Boa:
mynote = note(as_hz["C4"]) // use a dictionary of common frequencies mynote <- set "durationf" to dur
In this model, the instrument superclass will implement a standard protocol in which setting the duration attribute causes the instrument to set its own "gatef" attribute to the current amplitude (an inherited attribute, initially 1), and later set "gatef" to zero. "gatef" is passed on the the envelope by the note instrument.
In an composition system, you usually describe notes in terms of times and durations, but in a performance system, at least with a keyboard, you usually send start and stop messages, so you want to be able to create notes in this way too.
Nyquist:
(Nyquist does not have messages or even timed events.)
Ideal:
mynote = note(C4) // issue a timed message: mynote <- set "gatef" to 0.0 after 2.0
Boa:
mynote = note(as_hz["C4"]) mynote <- set "gatef" to 0.0 after 2.0 // or, you might also write: after 2.0 cause aura_set_to(mynote, "gatef", 0.0)
The point of this benchmark is to use something complex as a control or an update to a note or sound instance. If you create something complex, you need to control its lifetime, so automatic deletion is desirable.
Nyquist:
(defun note (pitch vib)
(mult (fmosc pitch vib) (env)))
(note C4 (lfo 6.0))
Ideal:
Boa:
class note (instrument): // instrument is superclass
def note(hz):
var e, o, v
// initialize fmosc with a block-rate modulator named
// b_zero so later, we can plug in a block-rate lfo:
output(mult(o = fmosc(hz, b_zero),
e = env()))
connect e to self // capture "done" messages from env
return self
on durationd: // pass durations to envelope
e <- set "durationd" to durationd
on gatef: // pass gate values to envelope too
e <- set "gatef" to gatef
on viba: // pass vibrato function to oscillator modulation
o <- set "moda" to viba
mynote = note(as_hz["C4"])
mylfo = lfo(6.0)
// assume that the lfo is to be deleted when the note is deleted.
// Instruments have a list of "owned" objects, so just add lfo to
// the list:
mynote.insert_child(mylfo)
In practice, we'll probably need to make a vibrato class so that we can combine an lfo with scaling and perhaps an envelope:
class vibrato (instrument):
def vibrato(rate, amplitude):
output(mult(lfo(rate), amplitude))
return self
Now we can add vibrato to note in a simpler style because vibrato will delete itself upon completion:
freq = as_hz["C4"] mynote = note(freq) mynote <- set "durationf" to 5.0 mylfo = lfo(6.0, freq * 0.01) mynote <- set "viba" to mylfo mynote.insert_child(mylfo) // transfer "ownership"
In this benchmark, the vibrato represents some quantity or control that is computed and is to be shared by many instances of notes or sounds. We do not want the control to be deleted when a note ends. Basically, this is a test of how flexibly objects can be interconnected and managed.
Nyquist:
;; create a global vibrato function:
(setf myvib (stretch 100 (lfo 6.0))
(defun note (pitch vib)
(mult (fmosc pitch vib) (env)))
;; we pass in myvib, a global, for note to use
;; rather than creating a new lfo instance:
(note C4 (lfo myvib))
Ideal:
Boa:
The implementation is just like before, except we omit the "insert_child()" call so that mynote will not delete mylfo. An issue that might arise is how do we delete lfo and what happens if we do. Aura will automatically take care of references to deleted instruments, replacing them with references to zero, so it is ok to delete mylfo at any time. How do we delete it after a specified duration? One possibility is to give it the standard instrument behavior, which is to send a "donea" message upon completion. Another, even simpler, way is to just send a delete message at the desired time:
mylfo <- set "deleteb" to true
This will be written in the future as:
mylfo.delete()
If mylfo sends a donea message, we could imagine a standard "helper" class called "destroyer" that we could use to delete any instrument instance upon completion. This scheme has good and bad points. The good is that the creator can arrange for disposal without remembering the object or creating any new methods. The bad is that it would be easy to forget to do this, and we need an extra object:
lfo_destroyer = new destroyer connect mylfo to lfo_destroyer // both are deleted when lfo sends its "donea" message // when lfo is destroyed, the connection to mynote is broken and // the input is replaced with b_zero automatically to avoid // a dangling reference
One more approach to deleting instruments is that we might make an option in the instrument class:
mylfo.auto_delete = true
Ideally, we should be able to use existing notes (from previous benchmark). It is not clear whether not sequencing should be sample accurate, block accurate, or asynchronous. I will assume block accurate; i.e. the next note starts on the block after the one in which the previous note terminates. Typically, we would expect the last samples of the first note to be in block N-1, so that in block N, we ask the note to compute again and it notices that completion has occurred and sends a "donea" message. Thus, we will be in the midst of a calculation when we discover it is time to start the next note.
This is an interesting benchmark because it implies some sort of temporal control structure: we are creating a sequence in time. The challenge for the implementation is to package up all the information needed to create the second note, probably without actually creating it. There must also be some way to sense when the first note completes so the second one can be instantiated and started.
Nyquist:
(seq (stretch 2.0 (note C4))
(stretch 3.0 (note D4)))
Ideal:
// add a helper function to pass duration to note:
def note2(pitch, dur):
var n = note(pitch)
n <- set "durationf" to dur
return n
// nyquist-like syntax would be ideal
seq(note2(C4), note2(D4))
Boa:
Can seq work as in the Ideal example? Let's see if we can actually implement it in Boa. This code will not delete the input instruments, but they could be set to "auto_delete" as in a previous example.
class seq (instrument):
var instr1, instr2
def seq(i1, i2):
instr1 = i1
connect instr1 to self
instr2 = i2
connect instr2 to self
output(instr1)
return self
on donea:
if donea == instr1:
output(instr2)
elif donea == instr2:
set "donea" to self // send completion message
This should be similar to the previous example, but we need to reuse a note object or prototype:
Nyquist:
(seqrep (i n) (note c4))
Ideal:
seqrep(n, note(c4))
Boa:
Again, we should be able to come close to the ideal. Here's an implementation of seqrep. This example illustrates why we don't want instruments to automatically delete themselves by default.
class seqrep (instrument):
var reps, instr, count
def seqrep(n, i)
reps = n
count = 0
instr = i
output(a_zero) // in case there are zero repetitions
self <- set 'donea' to instr // start iterating
return self
on donea:
if donea == instr:
if count < reps:
output(instr) // this resets the instrument
instr <- set 'trigger'
else:
count = 0
output(a_zero)
set 'donea' to self
This example brings up the issue of what exactly is a standard note interface. So far, we've seen the desire to start/stop a note by controlling the 'gatef' attribute, run the note for a given duration by setting the 'durationf' attribute, and now we want to run the note for its "natural" duration by setting the 'trigger' attribute.
The benchmark task is to play a note until the note completes. Then the computation should stop (i.e. it is not allowed to keep generating samples that are multiplied by an envelope that is zero.) It should be possible to reactivate the note to reuse the objects that have been configured.
The challenge here is to avoid hard-wiring how resources are used. Instead the user should have simple but powerful control over when resources are allocated and deleted. At the same time, it should be easy to manage resources correctly.
Nyquist:
(Nyquist does not reuse objects since there is no way to reference them.)
Ideal:
mynote = note(C4) play(mynote) after 10 cause play(mynote)
Boa:
The strategy here is to define AudioIO such that it disconnects instruments when they send a donea message. It should also break the message connection used to send donea. Now the client can write:
mynote = note(as_hz["C4"]) // create the note AudioIO <- set 'ina' to mynote connect mynote to AudioIO // to forward "donea" messages mynote <- set 'durationf' to 4.0 // start the note AudioIO <- set 'ina' to mynote after 10.0 connect mynote to AudioIO after 10.0 // play note again mynote <- set 'durationf' to 3.0 after 10.0
Maybe we can write a play function like in the ideal version:
def play(note, duration):
AudioIO <- set 'ina' to note
connect note to AudioIO
note <- set 'durationf' to duration
mynote = note(as_hz["C4"])
play(mynote, 4.0)
after 10 cause play(mynote, 3.0)
This benchmark tests the ability to compose sounds from simpler sounds. It should look like adding vibrato to a note (an earlier benchmark), where the vibrato object takes the form of a sequence of notes (another earlier benchmark).
It is one thing to have a built in sequence mechanism for notes, all of which run at the same level. Here, we are considering how well the language supports composition (in the programming language sense), where complex objects can be composed from simpler ones, and where control structures like sequence apply to micro as well as macro levels of the behavior.
Nyquist:
(defun note (pitch vib)
(mult (fmosc pitch vib) (env)))
(note C4 (seq (lfo 6.0) (lfo 4.0))
Ideal:
Boa:
Reuse the previous class defined in "Send Vibrato Function to Note (Delete When Done)"
class vibrato (instrument):
def vibrato(amplitude):
output(mult(lfo(6.0), amplitude))
after 1.0 cause output(mult(lfo(4.0), amplitude))
return self
Now we can add vibrato to note:
freq = as_hz["C4"] mynote = note(freq) mynote <- set "durationf" to 5.0 mylfo = lfo(freq * 0.01) mynote <- set "viba" to mylfo mynote.insert_child(mylfo) // transfer "ownership"
Some comments:
I considered two general approaches to dealing with Aura. One was to completely hide Aura within an object-oriented language. The would allow a very functional, nyquist-like, approach. The problems with this approach are:
The other approach (the one taken in this document) is to make Boa objects look like a subclass of Aura objects. This means you can reference these objects using Aura IDs, and you can, for example, set audio inputs of Aura objects to these "Boa" objects (because they are really Aura objects). The problems with this approach are:
mynote = note(