JACK Midi Control from your Android Phone over WiFi

Since my fundraising for my plugin GUIs is more or less bust, I've been thinking more about making music. I can't really stop developing, cause I'm fairly addicted, but I'm trying to ween myself back to just making whatever I want, rather than trying to develop stuff that will change the world. The world doesn't care that much. Anyhow, this blog is no place for bitterness.

But I have been playing with synths a bit more lately and still am really missing my expression controls. Now I could try to use the old touchmidi app I developed, but it only works with my laptop, and I now have a dedicated desktop in the studio to host my synths so I don't have a touchpad to use. I do have several android devices though. They should be great input interfaces. They can display whatever they like on the screen and have a touch interface so you can have arbitrary configurations of buttons, sliders, knobs, whatever. So I decided to figure out how.



There are a few tools you need, but first: an overview. The key to sending control information from one device to another is OSC (Open Sound Control), which in a nutshell is a way to send data over a network, designed for media applications. We need something to interpret touch input on the phone or tablet and send an OSC message. Then something needs to receive the message, interpret it, convert it to a midi message and send it out a JACK midi port. Well, we aren't inventing the wheel, there are several programs that can do these or related tasks.

One closed source system is TouchOSC. They have a free desktop app and the android app is a few dollars. But its closed source and doesn't actually work on linux. UNACCEPTABLE.
There are really several other apps, many of them free and none of them have had updates in the last few years. Ardroid is an OSC controller, but it is meant for ardour, you can't make custom interfaces like you can with TouchOSC.

What we want is Control. It hasn't been updated for a few years and its kinda buggy, but its OPEN. I could go fix it from the github source (but it would take me a bit of research on android stuff and I'm not a java fan) but its good enough as it is to get by. It uses customizable interface layouts written in JSON and you can add scripting to the interface through javascripts, so no crazy new formats. The real bugs are just in the landscape and portrait mode switching, so I have to open and close the interface a couple times before it gets it right. It's also cross platform.

I was able to make an interface after a few infuriating late night hours of trying to edit one of the examples (that I'm pretty sure had a lot of iOS specific JSON tags), then trying again next morning using the blank template and getting it done in about an hour. I never learn. Its a little involved to make a custom interface but there are good examples and decent documentation. It seems after a while of development the author focused much more on iOS and neglected the Android side, so there are several tags like color that don't work on android and make the interface much buggier if you attempt to use them. If someone wants to be super rad they would take this app and fix it up, make it a bit more robust and report json errors found in layouts etc... But its good enough that I'll just keep using it as it is.

For starters, you can just use the interfaces the app comes with. Go ahead and install it from google play. They have a pretty good variety of interesting interfaces that aren't all exactly conducive to midi control (remember apps CAN have built in OSC support like ardour and non-mixer do) but any of the interfaces built into Control will work. I'll tell you a bit about some other interfaces later.

UPDATE: Since I started writing this, another open source android app has come up: andrOSC. Its much simpler but allows you to edit everything in the app. Its a great option for anyone who wants a custom interface without any sort of file editing or extra uploading. Its a little too simple for what I want so I'm going to stick with Control for now.


Now we just need someplace to send the OSC data that the Control interface will output. Well I thought mididings was the tool for this job but it only accepts a few arbitrary OSC commands and there is no way to change the filtering through OSC aside from making scenes and switching between them. So that was a dead end.

But, the libraries used by mididings are openly available, so I figured I'd just make a python script to do the same thing. Except I've only edited a few python scripts and this is a few steps beyond that. The libraries used are actually c libraries anyway, so c wins. Hold on a sec, I'll just whip up a C program.

(Months go by)

Ah that was quick, right? Wait, Christmas is in how many days?!
Oh well. It works. And its completely configurable (and more features on the way). This new program, OSC2MIDI, gives you the flexibility to change the mapping wherever its easiest, in your OSC client app (i.e. our Control interface), in the osc2midi converter map file, or in the midi of the target program. In another post I'll describe how to use osc2midi more fully (with custom mappings) for now the default one works (because I wrote it to work with all the example interfaces that come with Control).

Download a snapshot, (EDIT: I just released the first beta, from now on, download the files, 2nd edit, updated link) extract them, and install them:
cd osc2midi
cd build
cmake ..
make
sudo make install

Then run:
osc2midi


Not too bad (if it didn't work cmake should help you, or just check the README for the dependencies). This gives you a jack midi client and opens an OSC host at 192.168.X.X:57120 (Your computer's IP address : Default port). If you don't know your ip address you can get it with the terminal command ipconfig. Connect the jack midi out port to Carla or whatever you like (I'll proceed with assuming you are using Carla).

Now we need to connect the Control app to the OSC server. Click the destinations tab and the plus sign at the top to enter in the address of the OSC host. From there go back to the interfaces tab and select whatever interface sounds like it will tickle your fancy. For this exercise we'll go ahead and use the mixer interface. If it looks squashed or doesn't fit on the screen you probably have to go back to the menu and open the interface one more time. I always have to open it twice like that (obnoxious, right? you should really go fix that!).

In Carla, load a plugin. Simple amplifier is perfect. First connect the osc2midi output port to the simple amplifier events input port. Open the editing dialog for the plugin and click the parameters tab at the bottom. Simple amplifier only has a single parameter (gain) and so you'll see a single horizontal slider and 2 spinboxes right of it. These numbers select the midi control number (CC) and channel.

Since we called osc2midi without arguments it defaults to channel 1 (or 0 if you prefer 0 indexing) and the first slider in the mixer interface on your android device will send midi CC 1 so select "cc #1" in the middle numeric box. You just bound that midi control to change the amplifier gain. If your jack ports are all set up correctly you should be able to change the first slider on your phone/tablet and watch the slider in Carla wiggle up and down with it. You have CONTROL!!!

The setup

 The other sliders are just sequentially mapped through CC 12. You can figure out what anything is mapped to by either using kmidimon to look at the midi output or running osc2midi in verbose mode (osc2midi -v). You can add other plugins and control them by the same method. Get creative with it.

For my next trick, we'll make some sounds with Control. Hit the menu button on Control and select the Conway's Game of Life interface. The default mapping works but I include a better one with the installation. So in the terminal hit Ctl+C to stop osc2midi. We'll start it again specifying a different map file. Type in:
osc2midi -m gameOfLife.omm
And the server will be restarted on the same port so you won't need to change anything else.

Now lets load an instrument plugin into Carla. I like amsynth but you can use any you like (though a polyphonic one will probably be more fun for this exercise). With an instrument plugin loaded just connect the midi-in port of the instrument to the osc2midi output the the audio output from your plugin/carla to your soundcard. Click several of the buttons on the grid in Control and you should hear some notes start playing. They will stay on till you click it again. Or click the start button and the notes start toggling themselves and you have a nice random sequencer.

This isn't quite enough for simulating all the expression you can get out of a good synthesizer or midi controller, but its a start. The best part is these open tools are all highly configurable, so we have a working system with just using all the defaults. But we can also setup very powerful and custom configurations by tweaking the interface in Control, the mapping in osc2midi, and often the midi bindings in your destination program (like Carla). But we'll save the rest for another post. In the mean time, try all the templates and get creative with these tools.