Good things come to those who wait, and, well, you all have been waiting a long time for this one. Version 3.0 of our Little MIDI Machine step sequencer app is now available in the App Store! Sorry it took so long!

Here’s an overview of some of the new features:

Virtual MIDI with audio background

This is a big one, it lets you sequence other synth apps on your iPad or iPhone with Little MIDI Machine! No longer do you need to hook up a hardware interface or a network MIDI connection to make some groovy synth music. Just load up AniMoog, Arctic Keys, iVoxel, NLogSynth Pro, Sunrizer or any other synth app that has virtual MIDI and audio backgrounding support. Since Little MIDI can sequence two channels at once, if you have an iPad 2 (or the upcoming iPad 3!) you can sequence two different synth apps with Little MIDI plus sync up a drum machine app, all at the same time. Sweet!

OMAC fast app switching

This feature works hand in hand with the Virtual MIDI support, it lets you assign a button on the main Little MIDI page to another app, so that you can quickly switch over to that app without having to double press the home key and go through the active apps in the menu bar. You can set which app to assign to which button (each sequence gets its own button) in the Settings menu. Currently there aren’t a ton of apps that support this, so if you don’t see your favorite synth app in the list, bug the developer to add this feature! Arctic Keys, Beatmaker 2, NLog Synth and ThumbJam currently support it, and it makes it much easier to move to those apps to tweak the synth parts being sequenced by Little MIDI.

Sequencer improvements

Sequencer improvements, oh yeah! The first big one was to add a second sequence to the iPhone/Touch version, to match it up feature-wise with the iPad version. Your iPhone can now sequence in both orange and green! Whoa, crazy colors, man…

Little MIDI v3 also now lets you sequence Note Length as well as Pitch and Velocity, so you can vary the length of each note in your sequences for more expression and feel. You can also now un-link the Pitch, Velocity, and Note Length parts of the sequence: this means you can set skip, random, and reverse separately for the three parts which can make the sequences much more interesting! For example, you could have a normal 16 note sequence of pitches, that uses random velocity values for each step, and a 13 step note length sequence so that the note length times are sequential but constantly varying with relation to the pitch. Cool!

Also added is a new super-sequence mode. This lets you gang together the 4 16 step sub-sequences (A B C D) into one big 64 step sequence, so you can have one nice long complicated sequence rather than 4 separate ones. You can also use this with the un-link feature described above in this mode!

New MIDI Input modes

Little MIDI now supports MIDI input to both record notes into your sequences, and transpose your sequences on the fly! Input record mode works when your sequence is stopped, it lets you enter in the note values of your sequence manually using an external MIDI keyboard or another CoreMIDI music app. Transpose mode works when your sequence is running, it takes the MIDI notes you input and transposes your sequence so that the note you enter is equivalent to the center slider value. This lets you, for instance, change the key of your sequence to match overall key changes in a song. Rad!

Improved MIDI configuration

Getting things set up and working is one of the biggest hassles of iOS MIDI, so we’ve overhauled the way you set up your MIDI configuration to make it (hopefully!) easier to use and more powerful. Easier on the eyes too! Check it out in the Settings Menu.

iCloud support

Little MIDI can now save your sequences in the cloud! If you have iCloud active on your iOS devices, any sequences you save to the iCloud area will be accessible to all of your other devices. This is especially useful in combination with the virtual MIDI feature: come up with a dope sequence on your iPhone at work using AniMoog (during, uh, lunch hour or something…), get home later that night and that sequence is right there on your studio iPad to sequence your Moog Little Phatty. Whoa, man! The future is here, like, now, man!

Other stuff

There’s more stuff too, like support for using a footswitch to start your sequences, the ability to clear your sequences with random data instead of zeroing it, a variable MIDI latency setting so you can reduce lag when you’re using virtual MIDI, and more.

And it’s still completely and totally free! Wow!! Why is it free? Well, basically because we’re idiots. Wait, no, I mean because we love you! Yes, that second one, free love, baby, yeah! And uh, okay, maybe a little bit of the first one too. Anyway, have fun with Little MIDI Machine v3, and please use it responsibly to make some groovy far out music! Whoa…

Little MIDI Machine v3.0, available now in the App Store!

Biophilia Educational Program in Iceland 2011 from Biophilia Educational Program on Vimeo.

Little MIDI Machine is being used as part of Bjork’s Biophilia education program to teach kids about music and science. You can see it being used at 2:10 in the video to control one of Bjork’s custom musical instruments.

“Here you can explore spatial and structural similarities in crystals and music, hear how viruses behave like generative music, see how the full moon governs the tide which then rations water into a music sequencer, there is a simon says method in learning differences in scales, DNA multiplying shows us how time signature and speed changes songs, an algorithm of gravity pulls a pendulum into natural counterpoint and the tectonic plates rub against each other and behave like chords.” -Björk

We are big fans of Bjork and are very interested in getting kids excited about music and science, so we were really happy to see one of our apps playing a small part in this very cool program!

If you haven’t already, be sure to check out her Biophilia app for iOS. It contains a bunch of interesting interactive music instrument apps based on her music, and also supports CoreMIDI so you can use the instruments to control your own synths.

Here’s our latest video, showing our FunkBox and Little MIDI Machine apps using an iRig MIDI interface and a bunch of other hardware and software to record an electronic music track.

To make our last two app videos we used an iPhone 4S to shoot the raw video. So far we’ve been very happy with the results, and we currently plan to shoot all our future videos this way. In addition to giving us some experience working with video on iOS, there is a certain appeal to doing as much of our iOS work as possible using the actual devices and their apps.

iPhone 4S nestled snug in the tripod

Previously we used a Canon S95 digital camera in video mode, so the transition was pretty easy. The S95 is a pretty basic camera, so while the iPhone camera and its app don’t have a lot of fancy features, for our purposes they work pretty much the same: point, light, focus, shoot. All we needed to do was find a way to get the iPhone mounted on a tripod, and then just swap the S95 out for the iPhone.

Glif tripod clip for the 4S

We used the Glif tripod clip to mount the iPhone on the tripod, which works great. The only downside is the iPhone can’t be in a case, so if you do use a case with your phone you’ll need to find one that lets you slip it in and out quickly and easily. Other than that, the iPhone fits in snugly to the Glif and works perfectly on top of the tripod.

We haven’t yet gotten to the point where we edit and publish the final videos on the iPhone or an iPad. For now we attach the iPhone to a Mac, import the raw video, and edit and polish things up with Final Cut Pro X. In the future we hope to do even more of the work on the devices.

My overly complicated hardware setup for my Boston Music Hack Day hack.

I recently spent the weekend of November 5-6 at Boston Music Hack Day. The idea behind these hack days is everybody who participates has 24 hours to come up with a “hack” project, build it and get it working, and then demo it to everyone else. 24 hours isn’t long enough to get anything super complicated done, and isn’t enough time to make things look pretty, but it is enough time to try out something new and cool. The beginning is always the fun part of any project, where you’re first brainstorming ideas, sketching the app out, and writing the “fun” code to handle the normal cases, so this seemed like a fun way to spend a weekend!

My hack involved getting songs on your iPhone/iPod’s music library to sync up with MIDI. I came up with this idea while listening to the Echo Nest talk about their database of music info, in particular the beat analysis data they’ve generated for millions of songs. I figured I could take their detailed information about the position of the beats in a song, hook that up to some of the MIDI clock sync code I’ve worked on before, and get something that would let you sync up a drum machine to any old song playing on your iPhone/iPod. Cool in theory, but would it work? By the end of the weekend I did indeed have it mostly working — some songs better than others, sometimes with some adjustments required — but my little app would play a song and send out a MIDI clock for it. Pretty cool! Here’s a quick video demo:

I spent most of my time working with the Echo Nest iOS client library. Under normal circumstances, I’d say this library is extremely easy to learn and use. Under the time pressure of getting something working within 24 hours, it was a little bit more stressful learning a new API and deciphering the data that came back from it! The basic process for querying and using the Echo Nest went like this:

Initialize Echo Nest library with my developer API key
Query for a track ID using the song and artist name of the music the user picks from their iPhone library
Use that track ID to retrieve a big chunk of beat analysis data from Echo Nest
Parse through nested structures of NSDictionary, NSArray and NSTimeInterval objects to get to the beat timestamps
Throw those timestamps into a simple consecutive array of times that I used to trigger MIDI clock messages

Once I had the data in a form I could use, I just needed to add the MIDI portion, which given my previous app work with CoreMIDI wasn’t too bad. Send a MIDI start message when the song starts, send 48ppq MIDI clock messages while it’s playing, and a MIDI stop message when the song stops. Receiving MIDI clock would have been a bit more complicated, so I left it out for this quick hack, but in theory that would be possible too so that you could slave the tempo of the song to your drum machine.

Finally I tested the now sort-of working app with various songs, and added a few tweaks to get it working better with some of the songs it had problems with. The tweaks were basically just allowing the user to manually shift the beat offset, so the bars in the song would sync up with the bars playing on the synced drum machine. These didn’t always automatically match because of weird little intro sections that didn’t match the rest of the song, or intro fade ins that only contained part of the measure, or various other reasons.

If I’d had more time, the first thing I would have done is improve these beat offset controls even more. I also would have liked to play with the pitch information that Echo Nest has embedded in their analysis data, so I could add a MIDI arpeggiator you could attach to a synth that would play appropriate not-terrible sounding notes for that portion of the song. Finally I would have liked to give the user the option to rearrange the parts of the song playing on their iPhone, so they could extend, repeat, or remove individual sections. All of these things would let the app become a full fledged MIDI remixer rather than just a syncer.

I have no plans for now to make this into a “real” app, but I could definitely see using some of this beat syncing technology in other apps like games or graphics toys that relied on the user’s music library. I’ll keep playing with the hack though, and if I can figure out a way to make it work a little better, and a little easier, maybe we’ll release it in some form.

A few of the other projects people worked on at the Hack Day that I particularly liked:
Bohemian Rhapsichord
Kinect BeatWheel
Grain Nest
Neurofeedback
Drinkify

And here’s a full list of all the hacks from the weekend.

Overall the weekend was a fun time, and I definitely recommend signing up for a Music Hack Day in your city if they have one! It’s a great chance to meet local developers working on music related things and a really nice way to force yourself to make something in just 24 hours. Thank you to @plamere and @elissab of the Echo Nest for organizing the Boston event!

FunkBox was included in the “Staff Favorites” sections of both the iPhone and iPad U.S. App Stores this week. This is the first time one of our apps has been picked for this feature, and we really appreciate it. Thanks Apple!

Looks like FunkBox was also included in an iPhone and iPad feature called “Retro Apps” this week! Really cool!

Welcome back to our series of blog posts about CoreMIDI. In part one we talked about setting up and playing around with CoreMIDI, and in part two we discussed the basics of how to add CoreMIDI to your app using Pete Goodliffe’s PGMidi sample project. This time we’re going to make some tweaks to PGMidi to improve MIDI timing, improve MIDI performance, and add MIDI clock features.

Before we start, be sure to check out the latest WWDC videos at Apple’s iOS development portal. There is one video in particular that discusses audio and music in Lion and iOS 5, including some bits about CoreMIDI, that you will find very useful.

Second, check out three CoreMIDI tips from Finger apps, makers of the first two apps to fully use CoreMIDI clock sync, MoDrum and BassLine. If these tips don’t completely make sense now, read them again after going through this blog post, since they are covering the same ground.

I also recommend buying and reading the draft version of Chris Adamson and Kevin Avila’s “Core Audio” book if you find you need help with figuring out how to query Core Audio/MIDI properties, and also just to get some good insight into Core Audio and MIDI in general. Good information on these topics is scarce, for whatever reason, but this book is really great and I wish it had existed (even in draft form!) when I was starting work on my music apps.

TIMING IS EVERYTHING

Okay, let’s begin with a quick look at how MIDI clock works. The basics of sending or receiving a MIDI clock are actually pretty straightforward. You have one master clock device that sets the tempo, and one or more slave clock devices that receive and lock to the master device’s tempo. When the master device is ready to begin, it sends a simple MIDI Start message that tells all the slave devices to start. It then sends a series of evenly spaced MIDI Clock messages at a rate of 24 per quarter note. The rate of these messages will tell the slave devices what the tempo is: each time the master clock sequencer clocks through a quarter note worth of song, it will have sent 24 of those clock messages. Which means each eighth note it would have sent 12 clock messages, each sixteenth note it has sent 6 messages, etc. In this way the master device is telling the slave devices both tempo and song/bar position: average tempo based on how fast the clock messages are coming in, and song/bar position based on how many total clock messages have been received since the start message.

If the master tempo changes, the master device will adjust the rate it sends its MIDI Clock messages by the appropriate amount, so the slave devices will notice those clock messages coming in at a different rate and will adjust their tempo also. Finally, when the master device stops its clock, it sends a MIDI Stop message to tell all the slave devices to stop playing.

So, to implement a basic MIDI clock you are going to need to send or receive three types of MIDI messages: MIDI Start to get things started, MIDI Clock messages at regular intervals to keep things going, and a MIDI Stop to stop everything. Easy enough, right? If you refer back to the MIDI reference page I mentioned in part 2, you’ll see that the actual MIDI packets are formatted like this:

MIDI Start: status byte 0xFA
MIDI Clock: status byte 0xF9
MIDI Stop: status byte 0xFC

You’ll notice that these messages are very simple and only contain the one status byte that identifies the MIDI message type. They don’t have any MIDI channel associated with them and don’t have anything identifying where they are coming from. When you send a Start message, you are telling everything connected to your MIDI outputs to Start. When you send Clock messages, you are broadcasting that same stream of clock ticks to everyone listening. And if you are the one listening, you assume only one device, the master tempo clock, is sending you this information. If for some reason two devices are both doing it at the same time on the same interface, you’ll receive twice as many Clock messages as you expect, and the raw MIDI packets will all look the same so your only option will be to go twice as fast.

TICKING AWAY THE MOMENTS THAT MAKE UP A LONG DAY

We talked about creating MIDI packets in part 2, and saw how Pete created note on and note off messages. Modifying that code to create MIDI Start, Clock, and Stop messages is pretty easy:

PGMidi creating MIDI packet data for note messages

Modify that to create a MIDI Start message

...or a MIDI Clock message...

...or a MIDI Stop message

You can put a very basic MIDI clock out in your app with just this. Send that MIDI Start, send 24 MIDI Clock messages evenly spaced in time for every quarter note, and finish it off with a MIDI Stop when you’re done. For now, you can just use a simple NSTimer to do the clock timing, or call it from your sequencer every 1/24 of a quarter note. For a quick first-try experiment, modify PGMidi’s sendMidiDataInBackground: method to run a simple clock in a separate background thread using sleep() like this:

Modifying PGMidi project's sendMidiDataInBackground to send MIDI clock instead

If you do this, you’ll note that the tempo on the receiving side should be approximately the tempo that you’re sending out. But you’ll also probably notice that the exact tempo on the receiving end, especially if you’re talking over a WiFi network connection, will be fluctuating back and forth by at least a few BPMs. This is due to a couple things.

First, there is some lag due to CoreMIDI and your MIDI connection. There is a variable amount of time between when you’re telling CoreMIDI to send those packets and when the other side is receiving them, because it takes time for the packets to be sent by CoreMIDI, then be transmitted over the physical connection, and finally to be received and get decoded on the other end.

Second, since you’re relying on the timing of the not-always-rock-solid NSTimer or sleep(), some of that variation could also due to your clock function not always firing off at the perfect consistent time you asked it to. This will especially be true when your app’s main thread is currently busy trying to handle UI or other events.

WHAT TIME IS IT? IT’S TIME TO GET ILL

A flaky clock is no good, we want solid steady timing. So how can we tighten this timing up? Luckily, CoreMIDI provides us a great solution in the form of putting timestamps on our MIDI packets.

CoreMIDI's MidiPacketListAdd() function

You’ll remember back in part 2 we discussed CoreMIDI’s MIDIPacketListAdd() function, which creates a MIDI packet list that is then sent by the MIDISend() function. MIDIPacketListAdd() has a time parameter of type MIDITimestamp. At the time, we mentioned if you just use a zero for the timestamp then the packet will be sent immediately. That’s exactly what PGMidi currently does. This is easy and works, but obviously the Apple engineers put that timestamp parameter in there for a reason. And happily enough, the reason they did it is to solve our current problem.

Using the timestamp parameter, we can hand off our MIDI packets to CoreMIDI a little bit early, with a timestamp set to a little bit into the future. This means the CoreMIDI code gets the packet ahead of time, giving it some advance time to handle it properly and get everything ready early, while still knowing exactly when you want that data to be sent. CoreMIDI can then hand the packet off to another app or the Camera Connection Kit or a WiFi network at exactly the right time.

From your perspective, this means if you are willing and able to give CoreMIDI your MIDI packets a little early, you will in return be rewarded by them being sent with extremely accurate and consistent timing. Which means you will get a clock that is seen as very steady and accurate by the receiving MIDI device.

That sounds great, right? But there was that little catch about sending those packets early…

Which leads us to an important question: exactly how early do we need to send these packets? Unfortunately, the answer to that isn’t going to make you very happy because it’s one you’ve heard many times before, usually when things are about to start getting ugly. “It depends”.

WHAT TIME IS LOVE?

One of the nice things about CoreMIDI is that it abstracts away the actual MIDI connection and hides the specifics of how packets are physically transmitted to and from you. Well, okay, most of the time that’s true; we’ll discuss some important but odd little differences with inter-app MIDI communication in part 5. For now, though, we’ll pretend that CoreMIDI lets you treat a MIDI connection the same whether it be connected via USB, over a network via WiFi, or another app running on the same device. Just hand your packets off to CoreMIDI and let it magically deal with all the details.

In theory this is great, it means you don’t generally need to worry about what sort of connection you’re using. You just focus on sending or receiving MIDI to and from various identical ports. The place it breaks down is unfortunately the same place we find ourselves right now: trying to figure out how early we need to send things to properly account for latency, so that our packets arrive right on time, every time. The reason it breaks down is that different connection types will require very different buffer amounts to account for their connection latency.

Sending MIDI packets to another app running on the same device requires extremely little latency. The next time that app gets a slice of processing time, which unless you are locking up the device should be very soon, it will be able to use that packet. We are talking on the order of single digit milliseconds. It is close to the amount of latency you’d expect between different parts of your own app.

Sending MIDI packets out the USB port is a little trickier, but should still be quite fast. I admit the nitty gritty details of USB driver latency are not my area of expertise. However, on the music making user side, both with iOS and on the Mac, I have personally experienced some flaky and jittery behavior from USB MIDI interfaces. It’s pretty safe to assume this connection will require more of a buffer than app to app. I would still guess it would be in the low 10-20 millisecond range at the most, though.

Sending MIDI packets out over the network via WiFi is the worst case. It just flat out sucks, and there is no point in sugar coating that. What makes it particularly terrible is that it is so variable: one day, on one network, you could send things 50 milliseconds early and it will work perfectly, rock solid. The next day on that same network using the same devices with that same latency value, it will be horrible and everything will sound noticeably off. I have personally found that if I up the latency buffer to around 200ms or so, it is pretty reliable almost all of the time. That is a lot of latency though, and only certain types of music applications (specifically, sequencers and clocks) will be able to send things that early.

Querying CoreMIDI for latency amount of a MIDI endpoint

So having said all that, what value should you use? Well, the first approach you should take is to ask CoreMIDI. As Finger apps mentioned in their CoreMIDI clock tips, each connection has a property named kMIDIPropertyAdvanceScheduleTimeMuSec that you can query which will tell you the latency value it thinks you should use. With WiFi networks, it will (as of iOS 4.3) always return 50ms. This is why you see the MIDI setup pages in BassLine, MoDrum and Molten add 50ms of latency when you turn on network connections: they are querying CoreMIDI to see what latency compensation amount they need to use for each endpoint, and then they use the largest of those values as their overall latency amount.

Using a network MIDI destination in MoDrum sets latency at 50ms

That’s great, it gives us something to start with, but does it work in practice? Does CoreMIDI really know best? In my experience… ummm, not really. I’m not sure how they picked that 50ms value, but I’ve personally found that sometimes it really just is not large enough. My advice is to play with different values, at different times, on different networks, until you find something that works enough that you’re happy with it. I know it is frustrating to hear that, but it is what it is. You are going to have to abandon theory and bang your head against the wall of reality for a while if you really want it to work well.

For now, when you’re just starting and trying things out, you might as well give the 50ms (or whatever other value that query returns) value a shot. As an added bonus, this means you can pass the buck and blame Apple if it doesn’t work, which is always nice. Once you get things working, I recommend you experiment with different values under different conditions and figure out what works best for your app, and use CoreMIDI’s suggested value as a minimum value.

THIS COULD BE THE LAST TIME, MAYBE THE LAST TIME, I DON’T KNOW. OH NO.

So, we’ve decided we’re going to hand things off to CoreMIDI a little early to tighten up our timing. And we’ve started thinking about exactly how early we’ll need to send them for it to work right. So now we actually have to figure out how to do all of that. First let’s take a look at how we use CoreMIDI’s MIDIPacketListAdd() function in the first part of our PGMidi sendBytes: method:

PGMidi's sendBytes method

As you can see and as we’ve discussed before, PGMidi is currently using a 0 for the timestamp when it hands its MIDI packets off to MIDIPacketListAdd(). As we’ve talked about, this means they will be sent immediately. All we have to do to queue things up earlier is give it a real time instead of that 0. Let’s quickly modify sendBytes: and make a new method called sendQueuedMidi: to do just that:

Our new modified sendQueuedMidi: method

A quick and easy modification to give us our timestamps: just add another parameter to the method that allows the caller to attach a timestamp to their MIDI packet, and then pass that MIDI packet on to MIDIPacketListAdd() which will do all the hard work of scheduling it for you. Done and done! Well, okay, almost done. What exactly were we supposed to put in that timestamp parameter again? UInt64, you say, eh? Okay, well, exactly what sort of UInt64 then, and how do you calculate it?

The simplest answer is that when you call the system function mach_absolute_time(), iOS will give you the actual current time in a UInt64 form. Yes, the same sort of UInt64 that this timestamp is looking for. Essentially you could replace the 0 timestamp in PGMidi’s original sendBytes: with a mach_absolute_time() timestamp, and you would get the same result: the packet would be sent immediately, at the current time. The question then becomes how to convert your sequencer’s time format to this other time format…

CoreAudio's AudioTimeStamp data structure

When writing an audio app, you deal with a number of different ways of keeping track of time. As shown in the AudioTimestamp format above, CoreAudio will give you several of the formats in every audio unit render callback, and you likely already use at least one of those in conjunction with your sequencer code. In order to translate between the type of time you use in your render callback and the UIInt64 time, you will want to use that mHostTime format that the AudioTimestamp provides.

For more detailed info about audio host time, check out the timestamp link from Finger’s CoreMIDI tips: Apple’s Technical Q&A QA1643.

One thing to remember is that an audio unit render callback is giving you the timestamp for the audio data it is requesting, which is a little bit ahead of time. It tells you that those audio packets will be played at that time, but you want to use mach_absolute_time() to know the actual current time. Another thing to remember is that you do not want to be making your CoreMIDI calls directly from that audio unit render callback, since it is running in its high priority Core Audio thread with tight deadlines. I’m only using the timestamp format used in the render callback as an example, since your sequencer timing code should be interacting with it already, and you should have some idea of how to translate between the two. Please don’t call these CoreMIDI functions directly from that Core Audio render callback after grabbing the time there, that’s very bad!

So hopefully you should be able to figure out how to translate your sequencer’s time units into CoreMIDI timestamp units without too much fuss. We also talked earlier about deciding how big you will want to make your MIDI latency buffer in milliseconds. So we also need to convert milliseconds units to this UInt64 format. This conversion will be a little trickier than it might seem, since the UInt64 can change depending on the platform/processor/etc you are running it on. You probably have already dealt with this sort of conversion in your sequencer or audio code. If not, here’s a link talking about how to do the conversion:

How to convert mach_absolute_time to milliseconds

So, the first thing you need to do is decide how early you want to send packets. You do this by querying the CoreMIDI interface latency amount property, by letting the user set it, or just through your own trial and error experiments. You will then have a value in milliseconds for your buffer, since that’s how normal people keep track of time. You then convert this millisecond value to a UInt64 amount using the conversion above.

Next, you want to convert your sequencer’s timing format to this same UInt64 format, if you aren’t already using it. You will have to figure out how to do this based on how the timing in your sequencer is set up. You want to know the time, in audio host UInt64 form, when the first equivalent audio packet to your MIDI packet will be fed to the render function. For example the time of the first audio packet in a note is equivalent to when you want to send a MIDI note on, so that is the time you need to calculate in audio host UInt64 form to know when to tell CoreMIDI that the note should be sent. Again, you are probably already doing something like this so you can tell your audio unit exactly when to start playing a sound, but you might not be using the audio host UInt64 as your base time reference format. You’ll need to either convert to and from it, or change your sequencer code to also keep track of and use it.

Finally, you calculate when you will need to give the packet to CoreMIDI by taking the actual time that event is supposed to trigger, and subtracting your buffer amount, since you want to give it to CoreMIDI early. This will be the point in time when you will hand off the MIDI packet to CoreMIDI, a bit before the actual event so it has time to process it. When the actual event occurs, your audio event will trigger and CoreMIDI will handle the MIDI portion based on the timestamp you gave it.

To summarize:

1. At time X you give your MIDI packet to CoreMIDI, with a timestamp of X+Y, with Y being the buffer amount
2. At approximately time X+Y, you will be doing your equivalent audio processing in your render callback for that event
3. At approximately time X+Y, CoreMIDI will also be sending your MIDI packet out

To get back to MIDI clock, as mentioned before you will be sending out 24 clock messages per quarter note. You want to send them out early by the amount of your latency buffer, and put a timestamp on each one. Ideally, your sequencer will have a timing resolution small enough to send these messages out individually. If your sequencer’s smallest resolution is greater than 24PPQ, you have a couple options: increase the timing resolution, or send out multiple messages at each of your smallest ticks.

The second option will work perfectly fine, you just queue up multiple Clock messages using MIDIPacketListAdd(), tagging each packet with an equally spaced timestamp, and then send them all out with MIDISend(). If for example your smallest clock resolution is a 1/16th note, which is 1/4 of a quarter note, you would then queue up 24/4 or 6 MIDI Clock messages at a time. You would calculate out the theoretical delta time between those packets based on a 1/96 (1/24 for each 1/4) note at your current tempo, we’ll call that Z. Given our X and Y times as described above in the summarize section, at time X with buffer Y (meaning you want that first Clock packet to be sent by CoreMIDI at time X+Y) you would queue up and hand off six MIDI Clock packets with timestamps X+Y, X+Y+Z, X+Y+2Z, X+Y+3Z, X+Y+4Z, and X+Y+5Z.

You hand off multiple Clock messages to CoreMIDI ahead of time, it sends each of them out at precisely the time you tell it to, resulting in the receiving device/app seeing an extremely stable MIDI clock.

TIME… IS ON MY SIDE… YES IT IS

MIDI Clock and MIDI Note messages are both MIDI messages, but they are obviously very different. Clocks are sent regularly with an identical packet format, Notes may be sent irregularly and have several fields such as channel and note value that will not be the same for different messages. You may find that you want to send your Clock messages and Note messages early by separate amounts to account for these differences. Clock messages require very precise timing, but are very regular and predictable, and their timing won’t change unless the tempo changes. Note messages, on the other hand, will need to be filled out depending on how the user modifies the note sequence, and how early you want to send those notes depends on how willing you are to tolerate lag versus how responsive you want to be to user changes.

If you use too large a latency when sending out MIDI Notes, the user will notice that the changes they make to the note sequences take a while to actually be heard, but the notes will arrive with perfect timing. A shorter latency value allows user changes to take effect more quickly, but the notes coming out may occasionally glitch. A good way to test this is to send out a long series of identical 1/16 or 1/32 notes under various conditions. You will quickly hear those glitches.

A large latency in clock values will be heard as a slightly lagged response to tempo changes, which depending on how dramatic those changes are might be much less noticeable. A shorter latency will again make jitter very apparent. In my experience, unless your app specifically needs to react to continuous and significant tempo changes, you can use much larger latency values on the clock to ensure it is extremely steady without losing too much in responsiveness. As mentioned before, I’ve used values of up to 200ms for network WiFi situations, resulting in a very stable clock even under terrible conditions.

For a third case, one thing that we can’t send early is anything that happens in real time: notes played on a virtual keyboard, real time taps on a drum pad, control data from a finger drag on an X-Y pad, etc. All of these happen in real time after being triggered by the user, and since we can’t reliably predict the future, we can’t know what the user is going to do ahead of time, and can’t send these messages early. They should always just be sent immediately, and if they do lag, there’s literally nothing that can be done about it.

This can also be a problem for our clock and sequenced data if they are affected in any way by real time user input. A primary example would be adjusting a slider that changes the tempo. In this case, we have clock and note data that needs to be sent early to correct for connection lag, but the data is being directly affected by real time input. Again, it is impossible for us to predict the future to know when the user will change the tempo. The end result is the tempo change will end up lagging by the size of your buffer. You will need to decide how important it is to track real time tempo changes versus how important it is to have rock solid consistent timing when setting the size of the buffer.

One very nice thing about CoreMIDI’s event handling is that you can send it multiple packets at different times with timestamps that are out of order and it will sort everything out for you. So if you are up for dealing with the complexity on your end, you can send it your clock messages 250ms early, your notes 100ms early, and your control data immediately as it happens, and CoreMIDI will sort all the timestamps out and send things out to their destination at their proper time, no matter when it received those various types of packets from you.

One last tricky thing to tackle when dealing with our MIDI clock is how to handle starting things up when we use large latency buffer values. For example, let’s say we have a clock buffer of 100ms, a tempo of 120BPM, and the user hits the start button on our app. We have two choices here, we can either:

The first option is the easiest, but with large latency buffers on the order of 100ms-200ms the user isn’t going to be very happy if they are trying to do something like manually sync the app to the sound of a record playing. On the other hand, we’re the master clock so if they are only trying to sync to connected MIDI slaved devices, this approach will work fine.

The second option is a bit more complicated, because it requires you to handle your clock in two separate phases: the startup phase where it’s catching up to the buffer by sending MIDI messages in real time or close to real time, and the normal phase where it’s sending out clock messages with the regular latency buffer. During the startup phase, we are not buffering by the full amount so the clock might be a bit jittery, however since this phase won’t last long it may not be a big deal. This is the approach I’ve taken with my apps, because I have DJ users who do want to manually beat match along to separate un-synced audio tracks. Having a 100-200ms delay may also just be a bit disconcerting to musicians who are sensitive to timing. Again, a latency vs. solid timing tradeoff will need to be made based on your particular app.

I BELIEVE IT, AND I BELIEVE YOU’RE RECEIVING IT

By now you should have a pretty good idea about how to set up your app to send MIDI clock and be the clock master device. Next, we need to figure out how we are going to receive MIDI clock so that your app can slave to the clock of a sequencer running on a laptop, an external drum machine, or another app. Again we’ll start with a very basic implementation, figure out where it falls short, and then think a little bit about how to improve it.

You will receive incoming MIDI Clock messages in your MIDIReadProc callback function, as described in part two. Per the MIDI spec you should expect to receive 24 MIDI Clock messages per quarter note, so you’ll be receiving a lot of them. As we covered above, these are very simple messages, they are just ticks or pings with no real content in the message itself. What matters to you is when they arrive.

Note that you will probably want to process these MIDI Clock messages differently than other messages since there are so many of them. At 120bpm you’ll be receiving 48 clock messages a second, which is a lot of messages. As we’ve mentioned before though, the messages do not carry any interesting information, you are only interested in the fact that this “ping” has been received at a certain time. All you will need to do is something simple and quick to keep track of them being received, and then you can discard the message.

For example, for a first-try basic implementation, whenever you receive a new MIDI packet in your callback function, instead of doing any complex processing just check the status byte to see if it’s a MIDI Clock message. If so, increment a count of how many of the MIDI Clock messages have been received. If the packet isn’t a MIDI Clock message, you’ll also probably want to check whether it’s a MIDI Start of MIDI Stop message, and only increment the MIDI ticks when you’ve received a start but not a stop. If the packet isn’t any of those messages, forward the packet as you normally would to your MIDI input processing function on another thread.

Elsewhere in your code, where your sequencer does its magic of deciding when to send the next note or calculate tempo to sync your LFOs, check that simple tick counter to see if you need to increment another step in your sequencer. Depending on the lowest resolution/granularity of your sequencer, this could be every 3 or 6 or 12 clock messages. Once you’ve passed that threshold, increment your sequencer another tick and trigger a new note, re-calculate your LFO speed, etc.

If you try this, you’ll find yourself on the other end of the jitter/latency problem we talked about above: your sequencer will increment properly in time with the external clock, but it will likely be to some extent unsteady depending on how long it is taking CoreMIDI to hand each of the Clock messages off to you. If you are talking to another CoreMIDI iOS app that is buffering its clock messages early, you might actually notice this jitter less because the CoreMIDI code on the other side may be handing the CoreMIDI code on your side the packets a bit early, helping to work around the problem! Remember that when you are picking a buffer in the send direction: you can actually improve other apps performance and help them out by choosing a larger buffer.

The bottom line is we need to figure out some way to prevent timing glitches caused by MIDI Clock packets that for whatever reason are given to us a bit late. If an individual packet is delayed, there is no way we can process it before it arrives, since that is physically impossible. So we need to find some way to smooth these incoming Clock messages and anticipate them, even if they arrive too late to trigger our sequencer properly.

Fortunately, with Clock messages we have some good options to accomplish this smoothing. As we’ve mentioned before, the clock is a regular and somewhat predictable series of ticks, rather than an irregular, unpredictable message to turn a given note on or change the value of a controller. We can’t accurately predict what the next note the user might press and when they’ll do it, that is impossible. But, we can make a guess about when one of those many MIDI Clock messages will probably arrive based on the current tempo, and if it turns out we were wrong, that just means the change in our tempo will lag a little bit.

Smoothing the incoming clock data will involve keeping a running averages of the tempo and song position to try to predict future clock messages, and firing off clock events to your sequencer whether or not a clock message actually arrived when you expected it to. You can think of this as having two separate timing engines: the first is driven by incoming clock messages, and uses the average timing information of those received ticks to output its own steady clock. The second engine is your sequencer, which uses that steady output of the first engine to increment itself step by step. Depending on your implementation, this will likely require keeping a history of received clock messages, including the time when they were received, and using that to calculate the running averages and predict when the next MIDI Clock message should arrive.

One nice thing about CoreMIDI is that incoming MIDI messages contain a very accurate timestamp of when that MIDI message was received at the lowest software driver levels. This allows you to make more accurate timing calculations even if the MIDI packet wasn’t immediately handed to your callback function by CoreMIDI. So one approach to predicting future/current MIDI Clock messages would be to use those very accurate timestamps from earlier packets, plus a running average of the normal delta between Clock packets, and then calculate when each future packet should arrive. When a packet “should” arrive, you tell your sequencer that it did, even if you don’t know for sure if it has yet.

Of course, even the lower levels of CoreMIDI might occasionally receive a packet with an oddball timestamp that throws everything off, for example if the master clock device or the physical connection has a glitch. Since outliers can negatively affect the calculation of an average, you may want to detect and throw them out, using a median approach rather than a strict mean, or just discarding data that is outside the normal bounds you would expect. In general though, your approach will be to keep a history of past packets, and use the average time received and average time between packets to guess when the next packet “should” arrive, and run your internal clock on this model rather than actual received packets.

Averaging methods used to smooth incoming jittery data will rely on predicting current and future packet times based on past packet times. This should work well in a situation where you have a stable, steady clock tempo coming in. The primary drawback is that if the master clock actually is trying to change the tempo, then you will lag those changes a bit since your algorithm will initially assume the tempo change is jitter and try to smooth it. How well you smooth your incoming data will directly affect how much you lag incoming tempo changes, so once again you will need to experiment with different buffer amounts to examine the trade off and determine the best behavior for your app.

Your final challenge will be getting your app to send MIDI clock out while slaved to a MIDI clock in. In theory this should be easy: just forward on any MIDI clock message that you receive. Unfortunately, due to these latency issues we’ve been discussing, that just won’t work: we will have a jittery clock on the receive side unless we smooth it out, combined with a jittery clock on the send side unless we send early. Whether you want to handle this case is up to you. I speak from experience when I say it’s tough to get right, in truth I haven’t really gotten it completely right myself yet. Essentially you will need to be buffering things on both ends, which depending on how you have things set up may or may not result in weird unpredictable results in practice, particularly when the app is first starting up the sync process. Give it a try once you’ve got everything else figured out. If you can get this one working reliably under tough variable latency conditions, you win the iOS MIDI clock grand prize.

IT’S TRICKY, IT’S TRICKY

Timestamps were one big step towards improving the performance of our MIDI processing. You might still be experiencing intermittent lag and latency when the user interacts with the UI. Try this out by moving a slider around while MIDI data is going in or out. You will probably notice that while you move the slider, your MIDI processing timing gets a lot flakier.

This is because so far we’ve been doing most of our MIDI processing in the main thread, which also handles UI tasks. So if the UI is busy doing something, such as processing a complicated touch event, our MIDI routines might be at the back of the line for processing time and noticeably lag. For some applications that might be okay, but for most it is not acceptable.

What we want is something that sits between CoreMIDI’s super high priority thread where we aren’t allowed to do very much, and the main thread’s medium priority thread that can get bogged down by UI and other tasks. To do this we will need to create our own thread with a priority in between.

A complete discussion of using threads is beyond the scope of this article. Here are some links to some information about how to set threads up, run code on them, and the sorts of pitfalls that await you when you try to do that:

Apple’s documentation on multi-threading
Where can I find a good tutorial on multi-threading in iOS?
How to use performSelector:onThread:withObject:waitUntilDone:
NSThread, NSTimer and AutoreleasePools in an iPhone SDK application
Threading priorities and abandonment
NSThread with _NSAutoreleaseNoPool error
Apple NSThread Class Reference

Also note that the multi-thread approach I will describe below is based on code I wrote for FunkBox, which has a minimum iOS version of 3.0. If you are writing a newer app with a minimum iOS requirement of iOS 4.0 or even 5.0 (I envy you!), be sure to read up on blocks and other fun new ways you might be able to do the same thing easier and more safely.

Disclaimers aside, I’ll now cover what we need to do for a simple MIDI thread. First, you need to allocate and initialize an NSThread. Next you need to give it a run loop, in which you need to pay particular attention to setting up an autorelease pool. You can find an example of how to some of this stuff here, again the subject requires much too much depth for me to cover adequately in this article.

Next, you want to set the threadPriority nice and high. The whole reason we are doing this is to try to help prevent the UI functions from hogging up processing resources and denying our MIDI processing a chance to run, so the priority needs to be higher than the main thread that is handling those UI functions. Keep in mind when putting things on the MIDI thread, though, that since they are running at a higher priority they should be relatively efficient and not hang up, otherwise you could get the reverse situation where MIDI processing adversely affects your UI performance. On a UI-centric platform like iOS that would be bad, particularly if the data that’s bogging you down is not important, for example parsing through large amounts of MIDI controller messages on a channel you aren’t even interested in. Thread priorities go from 0.0 to 1.0, you’ll want to aim for the high end of the range with this thread.

Finally, if you have never worked with threads before, a healthy amount of caution and even fear is in order. That doesn’t mean don’t use threads, it just means you should always be on your toes and be paranoid that they might be contributing to that weird sometimes-occurring problem that causes your app to crash. Be especially careful with what variables you create, release, and change inside your threads, as this will generally be the root of your problems. When you are working with threads, expect that things will get processed in unintuitive orders and ways. And when writing code that will be run on this thread, think very carefully about how it will interact with all of your other code.

When in doubt, try to treat it like a slightly more forgiving CoreAudio render callback: do as little as possible, as quickly as possible, as safely as possible, and then get out.

Okay, we now have our own nice custom thread up and going! We do still have to be a little careful using it, but it will help us out a lot as far as improving MIDI processing responsiveness. So how do we get the MIDI stuff running on there? Going back to Part 2, you’ll remember that we hand MIDI input processing off to the main thread using performSelectorOnMainThread:. We’ll use a similar method to run things on our own custom thread, but instead using performSelector:onThread: with a reference to our MIDI thread. We will want to do the same thing on the send side, within our sequencing code, when we have notes or clock data to send out.

Using performSelector:OnThread to run things on your MIDI thread

Now try the slider again and you should notice that MIDI performance will no longer be adversely affected when you move the slider continuously, since MIDI data is now being handled in a separate thread with a higher priority. Obviously this could potentially have some drawbacks as well: if you are sending a lot of MIDI data (several continuous controller values continuously cycling through values, such as with a fast LFO for example) the responsiveness of your user interface may suffer. As you’ve had to do throughout part 3, you’re going to have to experiment to figure out how much MIDI processing you can get away with while still having acceptable UI app performance. And always keep in mind that things will perform differently on different devices, so be sure to test this on your oldest, slowest device as well as your fastest one, because things that seem to work great on one of them might not work as well, or at all, with slightly slower or faster timing.

TRICKSY MIDI, HE HATES US

Finally, here are a couple more CoreMIDI functions you might find useful in specific situations:

CoreMIDI's MIDIRestart() function

Sometimes things are going to get a little wonky. Devices have been plugged in or disappeared, network connections are popping in and out, notes are hung. You should in theory be able to handle most of this, but sometimes for whatever reason, things just stop working and it doesn’t really matter why or who’s to blame, they’re just broken. And you and your users just want everything to work again. I’ve found in these situations calling MIDIRestart() helps get things back on track. This is good to call if you have a MIDI panic button. It can be good to call if you’re connecting things or disconnecting things, just to make sure everything still works. In general, I just call it whenever I’m changing MIDI configurations, just to be safe.

CoreMIDI's MIDIFlushOutput() function

As we’ve discussed above, you generally want to send MIDI clock and note messages early. At some point though, you need to stop, and you might want to do that immediately. However, since you were sending things early with timestamps, you might have already queued up a bunch of CoreMIDI messages to be sent. MIDIFlushOutput() called on each active MIDI endpoint will, in theory, get rid of those so that you actually do stop when you want to stop. In practice, CoreMIDI appears to send out network MIDI packets early across the network so they are receive at the right time on the other end, so you won’t be able to flush anything that has already left your device. But you should be able to get rid of anything waiting to be sent.

NEXT TIME…

If you’ve implemented everything so far, your MIDI-enabled music app should be looking pretty good! You have improved its timing and performance, and can send or receive MIDI clock. For a lot of apps, this might be all you need to implement: most users will be perfectly happy with just these features. But, if you’re interested in adding some more cool stuff to let your app connect to other apps on other devices, or even run alongside and sync up together with other apps running simulatenously on the same device, you’ve still got some more work ahead of you. We’ll cover the first one by talking about how to set up network MIDI connections with other devices next time in part four.

Here’s our latest MIDI related video. This shows our FunkBox app synced up via CoreMIDI with our Little MIDI Machine app. They are both running on the same iPad at the same time. Little MIDI Machine is sending notes out to the Studio Electronics Omega 2 hardware synth, you can see it below the iPad, it is playing two different synth lines. FunkBox is playing the drum beats, and sending MIDI clock to my new Korg MonoTribe synth on the right to sync it up, using Korg’s SyncKontrol app running on an iPod Touch.

The multitasking background-audio-capable version of Little MIDI is currently being tested and will be released soon!

The iPad is in an Alesis Studi/o dock, which provides MIDI and audio ins and outs. It works great and really makes iPad music apps feel like actual music hardware.

Also coming soon for the devs is CoreMIDI part 3… really… any day now…

Hey there, welcome back to the CoreMIDI brain dump. The last article was about familiarizing yourself with how CoreMIDI is used so you would know what you wanted to do, and would be able to test it. Part two will be about actually getting some basic MIDI functions working in your app, which is really all you were interested in hearing about in the first place, right? Let’s get to it then.

Here’s what we want your app to be able to do by the end of Part 2:

• Set up a quick and easy USB and Network CoreMIDI configuration
• Respond to MIDI configuration changes
• Send basic MIDI notes
• Receive basic MIDI notes

Here is what we will not be covering in this post, but that you can think about as we talk about the basic stuff:

• MIDI clock, CC/SYSEX messages (some of that’s next in Part 3)
• Latency compensation and other performance improvements (also Part 3)
• Setting up multi-device configurations (that’s Part 4)
• Setting up multi-tasking configurations (that’s Part 5)

In order to get this basic setup working, we’re going to be studying Pete Goodliffe’s PGMidi sample project (with updates here here and here) to give us some real code to work with. Pete was the first, and as far as I know is still the only person so far who has been kind enough to release a working CoreMIDI sample project for iOS. If he hadn’t released it, I wouldn’t be writing these posts. You can thank Pete for being awesome and helping us all out by buying his new iOS game The Enigma Squares.

If you’re using Open Frameworks with your app rather than native objective C, and looking at the PGMIDI code has you a little baffled because you’re not really a Cocoa/Objective C person, be sure to check out Michael Aldridge’s Sample Code to use CoreMIDI with Open Frameworks which will show you how to integrate PGMIDI with Open Frameworks!

Debugging network CoreMIDI with the simulator

Go ahead and grab the PGMidi sample project code now if you haven’t already, and get it running in the XCode simulator. The PGMidi sample app will enable the CoreMIDI network session when it starts up, so you will be able to debug the app using the simulator. You will need to run Audio MIDI Setup on the same computer and connect to “iPhone Simulator” as seen in the above picture.

Verify this works by also loading up the MIDI Monitor application on your Mac, and then send some MIDI notes from Pete’s app running in the simulator by clicking on the “Send MIDI” button in the lower right corner. You should be able to see some actual MIDI Note On and Note Offs messages that were sent from the simulator being received by the Mac. Success! Now we’re ready to kick some ass! Let’s get down to business and…

FREEZE FRAMEWORK

…go back to Apple’s CoreMIDI Framework Reference documentation. “What?!” you say, “but things were just getting cool?!”. Sorry Daniel-san, it’s time to go back to painting the fence.

In Part 1, I mentioned that this documentation is not that great to learn from. I still think that is true, it’s one of those documents that makes much more sense once you already mostly know what it’s trying to tell you about. As its name clearly states, it is a good reference, but it is not a guide or tutorial. So let’s look through it and we’ll try to find the good juicy bits that we’ll later see Pete using in his PGMidi Project.

Various CoreMIDI Framework Reference parts

The stuff on the left is all related to configuring the CoreMIDI network session, so your app can send/receive MIDI over WiFi or Bluetooth. We’ll be looking at these some more in Part 4, but for now, Pete’s only going to use a quick method call to NetworkSession to enable it. So we can forget about these files on the left side for a while. If you decide to peek anyway, you’ll notice they are all nice familiar Cocoa style objects. Enjoy, because that’s the last you’ll be seeing of Cocoa.

The stuff on the right… well, MidiSetup.h sounds like something we’d want to use, but actually it isn’t. MidiThru.h looks cool but I’ve never used it. And none of it looks particularly Cocoa like, does it? It all looks pretty confusing when you start trying to figure out why you’d want to use it. Let’s just pretend we never clicked over there and ignore those files too for now, and hopefully forever.

That leaves only one documentation file, MidiServices.h kicking it by its lonesome in the middle. This file contains the majority of what we’re going to need to use for now. Sweet, eh? Just one file! Can’t be too hard to learn one dumb file! Let’s just click and take a quick glance through it and… uh, well then. Hmmm, there’s a lot of stuff there, isn’t there? Oh, and that was just the functions, there’s data types too? And constants? Okay, this all looks really useful but, uh, where do we start?

For now we just want to focus on figuring out how to do a basic setup, send simple packets, and receive simple packets. But it’s still not immediately obvious what parts of MidiServices.h we should be looking at to figure out how to do that, so I’ll help and point some of them out. Here are some of the important functions and types we’re going to be using to get started using CoreMIDI:

MIDIPACKET FEVER, IT’S DRIVING ME CRAZY

MIDIPacket data structure

The MIDIPacket data structure is the heart of what we’re doing here: sending and receiving packets of MIDI data. As you can see from the typedef, it’s a pretty simple structure: timestamp, length, and the actual data.

We’ll talk more about the timeStamp in Part 3, for now you only really need to know that it is the exact time you want the packet to go out on the send side, or the time it was received on the receive side. PGMidi just sets this to zero on the send side, which means “send it right now”, and ignores it on the receive side. It gets more complicated when we actually want to use it and have to convert to and from other time formats to do so, but for now we can just ignore it.

length and data don’t need much explanation, you’re looking at the raw MIDI packet data you want to send and the length of that data in bytes.

MIDIPackets are always sent and received as part of a MIDIPacketList, which as you would expect is a list of individual MIDIPackets. To create a MIDIPacket, you actually create the MIDIPacketList first using CoreMIDI’s MIDIPacketListInit() function, which will initialize the MIDIPacketList and return a pointer to the first MIDIPacket in that list, which you will then use. More packets can be created and added to the list by calling MIDIPacketListAdd(). If this seems a bit confusing, and it is, you can see an example of how to do this in PGMidi.mm’s sendBytes: method.

To figure out what to put in those raw MIDI packets, you want to go look at the MIDI Specification. This spec comes from a standards organization and costs money. Googling “MIDI Specification” will get you some free and easier to understand explanations of the same thing, I’m partial to this one but they all work and the spec is pretty much set in stone at this point so you don’t have to worry about conflicting revisions. These are MIDI specifications, not CoreMIDI specifications. They are describing the plain old 1985 style MIDI bits and bytes that were originally sent over wires and that CoreMIDI will send and receive in its data payloads.

Keep in mind that CoreMIDI is only concerned with getting chunks of data to and from a destination. It doesn’t care what’s in those chunks of MIDI, it doesn’t know whether they represent note ons and offs, controller messages, program change, etc, and it won’t help you format the packet data. If you hand it completely random and nonsensical data, it will happily pass it on. If the other side of your MIDI connection returns the favor, CoreMIDI will chuckle and dump that random data on you without a second thought.

Bottom line: CoreMIDI is only concerned with making connections, handling timestamps, and exchanging bytes, it is not involved with the actual MIDI data. You are on your own as far as packing up the bits that make the data you send out, and deciphering the incoming MIDI bits you are given. The MIDI specs will tell you how to do that.

SHOW ME MIDISEND() THE FLOOR

CoreMIDI's MIDISend() function

So we have our MIDIPacket and now we want to send it somewhere. Unsurprisingly, the function we use to do that is named MIDISend(). MIDISend takes three arguments:

port is a MIDIPortRef, we’ll talk about these more later but for now you just need to know that your application has two of them, an input port and an output port. PGMidi will create two of these for your app in its init() method, and you will always use that same output port whenever you want to send data. So when we are calling MIDISend() we will point it to the output port.

dest is a MIDIEndpointRef, this is an individual destination for MIDI data such as the Network Session, or a USB MIDI interface attached to the Camera Connection Kit. PGMidi manages these endpoints as the application is running, you might plug in a USB cable which will make a new endpoint available, or turn off the Network Session which would mean you would no longer want to send MIDI data there. The key takeaway here is that you will need to call the MIDISend function once for each of the endpoints that you want to send the data to.

*pktlist points to the MIDIPacketList to be sent.

I AIN’T NO MIDIREADPROC CALLABACK GIRL

CoreMIDI receive MIDI callback

Sending stuff went pretty well, now of course we also want to be able to receive MIDI data. This one is a little trickier because we don’t decide when we’re going to receive things, we need to be given this data as soon as it comes in, whenever that is. To do this we need to register a callback function when we’re setting up CoreMIDI, telling it what to call when it has data to give us. The format of this callback function is described in the CoreMIDI documentation in the Callbacks section under MIDIReadProc.

PGMidi will register its callback function when it calls CoreMIDI’s MIDIInputPortCreate() function to create a MIDIPortRef. Remember it’s going to create one input port and one output port for the application, the input port will be linked with this function so that any input is sent there. PGMidi uses a static function called PGMIDIReadProc for its callback, so if you look at that function, you’ll see where received MIDI data is sent first.

An important thing to note about this callback is that it is being called in a high priority CoreMIDI thread. Much like a CoreAudio audio unit render callback, you want to limit the amount of processing here, and avoid allocating and deallocating objects. Any significant processing will need to be done once you’ve handed off the received data to another function running in another lower priority thread.

CLIENTS AND ENDPOINTS AND PORTS, OH MY!

We’ve talked about sending and receiving packets, but before we can do any of that we need to configure CoreMIDI so it knows where it can send to and receive from. This is actually where PGMidi does most of its work, and that’s a good thing, because it’s one of the more confusing parts to figure out without a general overview of how things work.

Glancing through the CoreMIDI framework documentation you’ll see references to ports, endpoints, entities, clients, devices, and there are explanations given for all these, but it’s hard to figure out what’s what in the context of a regular old iOS app. This is because the CoreMIDI docs need to describe what each part can theoretically do for any application, even if a lot of those uses are esoteric or overly abstract from a simple music app developer’s point of view. From the Apple CoreMIDI engineers perspective this makes sense, they are thinking through every different type of use when they develop the framework and need to be sure they have covered every case, especially the unusual ones. From the perspective of you and me, the lack of simple and practical examples can often make it really difficult to figure out what parts are actually important.

Anyway, here’s the quick guide. Like your Physics 101 class in college, my goal here is to get you to have a basic understanding of what’s going on that you can use to solve simple problems, even if the reality is I’m oversimplifying things in places. From your point of view, this is what these things do:

MIDIClientRef will be created once when you’re setting up CoreMIDI. You create one of them, and always use the same one when CoreMIDI wants it to know who you are. Think of it as your app’s way of referring to itself when talking to CoreMIDI.

MIDIPortRef will be used to keep track of your two ports: one input and one output. Once you create your client, you then create your input and an output ports and associate them with the MIDIClientRef, so your app can send and receive MIDI data. Even if you are receiving input from multiple places, it will all come through your one port. Likewise for the output, you may be sending notes over USB and clock over WiFi, but it’s all associated with your output port.

MIDIEndpointRef is a more specific input or output, it is a specific USB interface or Network Session that you either send to or receive from. It is analogous to a physical MIDI cable that sends or receives 16 channels of MIDI information. You can have multiple endpoints coming in, multiple endpoints to send out to. These endpoints can also come or go as your application is running, if the user plugs in a new USB MIDI interface, for instance.

So let’s say you are sending MIDI data out to three different places: a drum machine attached to a USB MIDI interface, a laptop attached via WiFi, and to another app on the same device through a virtual port. Each of these three destinations will have a MIDIEndpointRef associated with them that you will use when calling MIDISend. Sending to one of the three destination will require you use your applications output MIDIPortRef, along with the specific MIDIEndpointRef of that destination.

MIDIEntityRef is a collection of MIDIEndpointRefs, for example a simple 2×2 USB MIDI interface will have a two inputs and two outputs, each of which is separate MIDIEndpointRef. The MIDIEntityRef owns all those endpoints, and is used to logically group them together. You don’t really care about this much for a simple app, but in some functions instead of asking for a MIDIEndpointRef, CoreMIDI will ask for a MIDIEntityRef. All you really need to know here is that if CoreMIDI wants a MIDIEntityRef, you can use CoreMIDI’s MIDIEndpointGetEntity() function and give it one based on your MIDIEndpointRef.

CALIFOR-NOTIFICATION

MIDINotification

MIDINotification data structure

The last CoreMIDI data structure we’re going to look at before getting back to PGMidi is MIDINotification. There isn’t much to see here in the structure itself, you just have the messageID and messageSize parameters that you use to figure out what the notification is telling you. The important thing here is that you will be receiving these whenever something changes: a new interface is added, a configuration is changed, an error occurs. When that happens, you’ll be passed a MIDINotification to your notification callback function, and you will need to handle it. That callback function is defined when you create your MIDIClientRef using CoreMIDI’s MIDIClientCreate() function, which PGMidi will take care of for you in its init() method.

You can see the type of notifications you’ll be sent by checking out the MIDINotificationMessageID constant list.

MIDINotificationMessageID constants

Alright, enough of that, we now have an idea about some of the lower level CoreMIDI data structures and functions we’ll be using. Let’s go back to Pete’s code and see how to actually put all these various parts together into a project that actually does something.

HEY PGMIDI YOU’RE SO FINE YOU’RE SO FINE YOU BLOW MY MIND

PGMidi MidiMonitor app

First of all, what is this project supposed to be doing? It is a basic MIDI Monitor that prints out anything that is received so you can test MIDI input. It will also let you send some random MIDI notes so you can test MIDI output. Finally it will set everything up and manage your MIDI connections so that you can react when things are plugged and unplugged. Most of the code is actually dedicated to that last part, setup and configuration, which is nice because it’s probably the part you care least about tweaking right now. The thing you probably want to be able to do at this point is get some MIDI in notes triggering sounds in your synth app, and have your virtual app keyboard triggering MIDI out notes, and you really just want enough configuration to let you do that.

So let’s take a look at the files in Pete’s sample project:

PGMidi Project files

You’ll be re-using everything here that starts with PGMidi. You’ll be using your own code in place of anything that starts with MidiMonitor.

As you might expect, the most important parts of Pete’s code are located in PGMidi.mm. This is where he is doing most of his CoreMIDI setup, managing connections, handling notifications, and sending and receiving MIDI data.

Before we dive into there though, let’s look at what MidiMonitorAppDelegate.m and MidiMonitorViewController.m are doing, since these are files you won’t re-use, but are the parts that interface with PGMidi in the same way you will need to in your project. By looking at them, we can figure out what you will need to add to your own app to get it working with PGMidi.

Initializing PGMidi in MidiMonitorAppDelegate.m

Looking in the didFinishLaunchingWithOptions: method of MidiMonitorAppDelegate.m, we can see that he first checks to make sure we’re using a version of iOS that can use CoreMIDI. If so, he allocs and inits the PGMidi object, calls a method to enable the network session, and then links it up to the viewcontroller, which is MidiMonitorViewController. So far so good! Easy enough to do a similar setup in your own app delegate or elsewhere.

Next we will look at MidiMonitorViewController.m, first noting that this class is both a PGMidiDelegate and a PGMidiSourceDelegate. The PGMidiDelegate functions will give this class notifications about sources and destinations being added or deleted. The PGMidiSourceDelegate function provides the mechanism for receiving incoming MIDI packets.

One thing you will notice is that the various methods of MidiMonitorViewController.m display everything using the addString: method. This just takes input strings and puts them in a textview. It has nothing to do with CoreMIDI, but it’s the end result of pretty much everything that happens in this class: received MIDI input, errors, notifications, etc will all end up being sent to that textview.

midiReceived:packetList delegate method in MidiMonitorViewController.m

Next let’s look that the midiReceived delegate method, which is where PGMidi is going to pass any MIDI data received from its CoreMIDI callback function. This is the function you will be modifying to do your MIDI input processing. Note that this method is being called in CoreMIDI’s high priority thread, so whenever it calls anything it makes sure to performSelectorOnMainThread to move the processing on to a lower priority thread, in this case the main thread. In Part 3 we’ll talk about setting up a dedicated MIDI thread for processing so that things like UI handling don’t cause hiccups in MIDI processing. For now though, we just want to move things out of that high priority CoreMIDI thread.

The other important part here is the for loop. It is processing the received packet list, packet by packet, and handing each one off to addString: to be printed. Unless you are also writing a MIDI monitor app, you will want to process these packets and figure out which are notes, clocks, etc instead of just printing them.

To do your custom processing, you will want to be looking at the raw MIDI information in packet->data[]. Again, you will want to do any significant processing in a lower priority thread, but you might want to peek at the data headers here to decide where to hand off the data. Individual MIDI clock messages, for instance, will be coming in so quickly you might want to process them differently than notes or control changes.

You can determine the type of MIDI data by peeking at the first byte in the packet, which will be in packet->data[0]. A value of 248 here will be a MIDI clock tick message, 250 will be a MIDI start message, etc. Anything more than a quick check of the header needs to be done in a lower priority thread.

Finally, note that while this MidiMonitor sample code does pass the incoming MIDI data off to a lower priority thread, the addString: function used to do so is creating a new string object in the high priority thread, which is probably not a great idea. It works fine but it’s not recommended to do that in this thread. The first code tweak you might try would be to replace this part by copying the incoming MIDI packet data into some pre-allocated memory and then passing it off to your processing function in another thread to be handled.

sourceAdded: from MIDIMonitorViewController.mm

We also see various PGMidiDelegate functions related to changing configurations: sourceAdded:, sourceRemoved:, destinationAdded:, destinationRemoved:. The heavy lifting of handling these setup changes is being handled by PGMidi, here we are just printing out some text about what happened. Do note, however, that when sourceAdded: is called we need to make ourselves the delegate of that PGMidiSource so that we will receive incoming MIDI input from it.

sendMidiDataInBackground method in MidiMonitorViewController.m

The last thing we’ll look at here is the sendMidiDataInBackground: method which is called in a background thread when you press the send midi button in the app. Here he is sending out 20 random MIDI notes. You’ll notice that each note has a note on and then 0.1 seconds later he sends a corresponding note off. Here you also get your first taste of how to create that raw MIDI packet data that you’ll be giving to CoreMIDI. No objects here, son, we’re packing together raw bits.

As mentioned previously this site is a nice reference for the format of those MIDI packets. If you’re going to be doing anything complicated, you’d best be getting your bitwise operator and bit shift skills honed and ready, because you’ll need ‘em.

PGMidi sends all MIDI packets immediately, which is why he is sleeping the background thread for 0.1 seconds before sending a note off message. To send MIDI notes using the standard PGMidi methods you’ll need to do something similar. In Part 3 we’ll talk about how to set the timestamps for MIDIPackets so you can queue things up ahead of time, and hand both the note on and note off to CoreMIDI at the same time. These timestamps are also useful for sending your MIDI packets early to overcome the latency of, for instance, a network connection.

You will be replacing this function with your own code that sends MIDI note ons and offs. Most likely your app won’t need to send 20 random notes, so when you modify this function the first thing you’ll want to do is get rid of that for loop. Next, replace the RandomNoteNumber() function with the actual note pitch you want to be sent. For reference, 48 is a nice middle C.

Another thing that can be tweaked is the MIDI channel number: here the 0×90 and 0×80 values are sending note on and note offs to MIDI channel 1. To change it to channel 4, for instance, you would use 0×93 and 0×83. Also, the velocity of the note can be tweak by changing the 127 in the noteOn[] const to the velocity value you desire.

Using this information, you should be able to easily rewrite this function to take a few arguments like note pitch, channel, and velocity, and send PGMidi some useful MIDI data packets instead of the random ones.

PGMIDI TREATS OBJECTS LIKE WOMEN, MAN

Let’s take a look at the PGMidi.h and PGMidi.mm files:

PGMidi, from PGMidi.h

Pete’s MidiMonitor app has one PGMidi object, it is created in the app delegate and manages CoreMIDI for the lifetime of the app. It creates a MIDIClientRef, client, that CoreMIDI will use to identify your app. It also creates two MIDIPortRefs, an outputPort and an inputPort, which are used to send/receive all MIDI packet data to/from CoreMIDI. It has a delegate that it will inform of changes to the MIDI configuration, when sources or destinations are added. And finally it has arrays of sources and destinations, which hold PGMidiSource and PGMidiDestination objects that manage your CoreMIDI connections.

PGMidi methods, from PGMidi.h

PGMidi has a few methods that can be called. You should call enableNetwork: right after initializing your PGMIDI object to enable the CoreMIDI Network Session, so you can send and receive data over WiFi or Bluetooth. There is really no good reason not to do this, especially if you are a sound source like a synth, since it will allow other apps on other devices to connect to you and send you MIDI data.

In Part 4, we’ll talk about how your app can look for other devices and connect with them. For now, by enabling the network session we at least allow other apps on iOS devices, and Macs running Audio MIDI Setup, to connect with us. You can verify that by running MidiMonitor on one of your test devices, and MoDrum on another test device. You’ll notice the device running the PGMIDI Monitor will show up in MoDrum’s MIDI Setup connection list as long as the PGMIDI app is running.

The other two methods in PGMidi are sendBytes: and sendPacketList:, these are useful to broadcast MIDI data to every available output. When you are first putting CoreMIDI in your app, if you don’t want to worry about configuring which outputs to send things to and just want to blast it, call these PGMIDI send methods instead of the ones in the individual PGMidiDestinations.

Peeking in PGMidi’s init function, you can see it first creating a MIDIClientRef and then setting up two MIDIPortRefs, inputPort and outputPort. Finally it calls the scanExistingDevices method, which will add all available MIDIEndpointRefs to PGMIDI’s sources and destinations arrays. We’ll cover this function in a little more depth below.

The remaining private methods defined in PGMidi.mm are used to handle notifications, and connect and disconnect sources and destinations. You shouldn’t need to tinker with any of these for now, since PGMidi will let you know what’s happening via the PGMidiDelegate protocol, and then you can just look through the sources and destinations arrays. At some point, once things are working and you’re feeling more curious, you will want to look through these methods to see how and when PGMidi is calling the lower level CoreMIDI functions to connect and disconnect MIDIEndpoints.

PGMidiDelegate methods, from PGMidi.h

PGMidi has four delegate methods we talked about before that are used to tell its delegate about sources and destinations being added and removed. As you can see above, they are named exactly as you would expect them to be. The part of your app designated as the PGMidiDelegate will need to implement these methods so that it can receive configuration change information from PGMidi.

Let’s take one more look at PGMidi’s init function in some more depth to see how it is setting up CoreMIDI for us, and the CoreMIDI functions it calls:

• 1. It creates a MIDIClientRef and sets up PGMIDINotifyProc as a notification callback function using CoreMIDI’s MIDIClientCreate() function.
• 2.It uses that MIDIClientRef to set up two MIDIPortRefs, inputPort and outputPort, using CoreMIDI’s MIDIInputPortCreate() and MIDIOutputPortCreate() functions. When creating the inputPort it also sets up PGMIDIReadProc as a callback function for incoming MIDI data.
• 3.It calls the scanExistingDevices method, which will call CoreMIDI’s MIDIGetSource() and MIDIGetDestination() functions to get all available MIDIEndpointRefs and set them up as PGMIDISources and PGMIDIDestinations.

Again we see the familiar MIDIClientRef, MIDIPortRefs, and MIDIEndpointRefs. Most of the CoreMIDI set up that PGMidi is doing just involves creating, destroying, configuring, or connecting these for you.

YES! NO! I DON’T WANT TO BE YOUR PGMIDI OBJECT

The three other PGMidi.mm objects are PGMidiConnection, which describes a basic MIDI connection, and PGMidiSource and PGMidiDestination, which inherit from PGMidiConnection and have specific methods designed for sending or receiving MIDI data.

PGMidiConnection base class, PGMidi.h

PGMidiConnection contains a link to the PGMidi object, a CoreMIDI MIDIEndpointRef for the connection, a string containing the name, and a boolean that tells you whether this connection is a network connection. Pretty straightforward stuff, you will need that MIDIEndpointRef when you want to send MIDI data, and you’ll use the name to sort through the different connections.

PGMidiSource inherits everything from PGMidiConnection and adds a delegate for received MIDI data. When the PGMidiSource object receives incoming MIDI data, it will then hand it off to that delegate using -(void) midiSource:(PGMidiSource*)input midiReceived:(const MIDIPacketList *)packetList, which is a PGMidiSourceDelegate delegate method.

PGMidiDestination inherits everything from PGMidiConnection and adds two methods, sendBytes: and sendPacketList:, which are similar to those same functions in PGMidi but in this case are used to send MIDI data only to this particular destination.

PGMidiSources and PGMidiDestinations are connected and disconnected using the connectSource:, disconnectSource:, connectDestination:, and disconnectDestination: methods. These will either be called by the scanExistingDevices() function during init(), or called when PGMidi receives a notification that a configuration change has occurred. If you look at connectSource and disconnectSource you will note that PGMidi is calling CoreMIDI’s MIDIPortConnectSource() and MIDIPortDisconnectSource() functions to associate (or disassociate) a given MIDIEndpointRef with the inputPort MIDIPortRef. The connectDestination and disconnectDestination do not need to do this, since you give the MIDISend() function a MIDIEndpointRef each time you send data.

The other PGMidixxx files in the project are tools to help find matching source/destination pairs if you’re trying to speak to one particular device amongst several available ones. For now, I wouldn’t worry about it but if you have some specific routing needs, check those files out.

SUMMARY OF HOW PGMIDI HANDLES SETUP, MIDI SEND, MIDI RECEIVE

Walkthrough of how PGMidi sets up CoreMIDI:

• PGMidi is alloced and inited in the app delegate, network session enabled
• PGMidi creates MIDIClientRef in init using CoreMIDI’s MIDIClientCreate() function
• PGMidi creates inputPort and outputPort MIDIPortRefs in init using CoreMIDI’s MIDIInputPortCreate() and MIDIOutputPortCreate() functions
• PGMidi creates PGMidiConnections with a call to scanExistingDevices: in init. These connections are based on CoreMIDI MIDIEndpointRefs
• After the initial setup, added and removed connections are handled with PGMidiNotifyProc notification callback that was defined in MIDIClientCreate() in PGMidi’s init
• The PGMidiDelegate (your code) is informed of configuration changes through sourceAdded: sourceRemoved: destinationAdded: destinationRemoved: delegate methods

Walkthrough of creating and sending a MIDI packet using PGMidi:

• Create raw MIDI payload data in your app using the proper MIDI format
• Hand that chunk of data off to PGMidi via sendBytes:
• PGMidi will put that data into a MIDIPacket data structure in a MIDIPacketList, using CoreMIDI’s MIDIPacketListInit() and MIDIPacketListAdd() functions
• PGMidi will call the CoreMIDI MIDISend() function with that MIDIPacketList, for each MIDIEndpointRef/PGMidiDestination in its destinations array
• MIDI data is sent out by CoreMIDI to its proper destination

Walkthrough of receiving MIDI input data using PGMidi:

• CoreMIDI calls PGMidiSource’s static PGMIDIReadProc callback function with a MIDIPacketList, in a high priority thread
• MIDIPacketList is passed to PGMidiSource’s non-static midiRead: function, still in that high priority thread
• PGMidiSource passes the MIDIPacketList to the midiRead: function in your PGMidiSource delegate, still in that high priority thread
• You then want to process each MIDIPacket in the MIDIPacketList, handing off the MIDI payload data for each packet to a function in another thread

WHAT YOU NEED TO DO

To summarize what you need to do to in order to get PGMidi working with your app:

• Add CoreMIDI.framework to your project, linked weakly if you are supporting iOS versions < 4.1
• Copy PGMidi, PGMidiAllSources, PGMidiFind files over to your project
• Add PGMidi setup code similar to MidiMonitorAppDelegate.m to your app delegate
• Add PGMidi delegate functions to the object in your project that is handling configuration notifications, similar to the sourceAdded: sourceRemoved: etc code in MidiMonitorViewController.mm
• Add the midiReceived: PGMidiSource delegate function to the object in your project that is handling MIDI input, modified from the midiReceived: code in MidiMonitorViewController.mm
• Create your own new SendMIDINote function that calls PGMidi’s sendBytes: to send out MIDI notes, modified from the sendMidiDataInBackground code in MidiMonitorViewController.mm

So that’s PGMidi in a nutshell. Once you’ve done all the above, your app should now have basic CoreMIDI support that enables it to set everything up, and allows it to send and receive basic MIDI notes. A great start!

POTENTIAL SAMPLE PROJECT

If you don’t have a good sample music app project to work with, you might try adding CoreMIDI support to Allen Porter’s open source mobilesynth project. Add some code so that received MIDI notes will trigger the synth and a tap on the virtual keys will send MIDI notes out.

NEXT TIME…

Next in Part 3: Timing Is Everything, we’ll talk about some ways to improve on PGMidi by queueing up MIDI packets early with timestamps, adding MIDI clock functionality, and creating your own MIDI processing thread so that UI functions don’t affect your MIDI timing.

I’m writing these to help celebrate the Palm Sounds blog’s fifth anniversary: five posts for five years of great coverage of all sorts of mobile music making. As a music app developer you want to keep on top of all the other latest apps being released, and get feedback on the sorts of features that users are looking for. Palm Sounds is a great place to do that. Happy 5th and thanks!

WELCOME TO THE COREMIDI THUNDERDOME

These posts are an introduction to adding CoreMIDI to iOS music apps for developers who have never used the CoreMIDI framework before. Some devs are lucky enough to find learning this stuff is really easy, and can pick it all up themselves very quickly without needing any of this kind of help. Sadly for me, I am not one of those people. Figuring out CoreMIDI was not easy and at points it was actually extremely frustrating. And it still is sometimes, even after implementing it in a couple apps.

So, if you happen to be like me and find this stuff pretty hard, then maybe these posts will be helpful! And if you’re not like me and CoreMIDI already makes perfect sense to you, well, then get out of here and go make some more cool apps. And stop making me look bad!

I LOVE IT WHEN A PLAN COMES TOGETHER

So here’s the plan for how we’re going to try to tackle the CoreMIDI beast in five short blog posts:

• This first part is a basic introduction to CoreMIDI, helping you figure out what it is you want to do, creating working MIDI music setups, and grabbing some apps to test your setup with.
• Part two will be about implementing basic CoreMIDI functionality using Pete Goodliffe’s PGMidi sample code, and will hopefully get you to the point where you can set up CoreMIDI, send and receive MIDI notes, and have a basic idea of what’s going on.
• Part three is when things start to get interesting. We’ll cover improving the performance and timing of MIDI by scheduling MIDI events in advance, sending and receiving MIDI clock, using a dedicated MIDI thread, as well as give you a couple other helpful techniques to avoid some specific headaches.
• Part four will deal with setting up network MIDI connections between iOS devices, and between an iOS device and a laptop, so you can find, connect and communicate with other apps running on other devices.
• Finally, part five will discuss what you need to do to allow your app to work in the background so that you can sync and communicate with other apps running simultaneously on the same iOS device.

What these posts won’t give you is a sample project full of code that you can insert into your own app to make all these features automatically work. I’ll try to point you in the right direction to implement everything, try to help make sense of things with links and advice, and help you avoid some of the many pitfalls and deadends I ran into as I worked on things… but… I will not be providing code you can just #include to solve all your problems. Sorry about that! Hopefully this will still be helpful.

INFORMATION UNDERLOAD

Good introductory information about CoreMIDI is currently pretty tough to find. Usually when you want to add a cool feature to your iOS app, you can Google up a bunch of tutorials, sample code, and online discussions that help you figure everything out. CoreMIDI, not so much.

Your first stop is Apple’s official CoreMIDI Framework Documentation, so go ahead and check that out now. Makes perfect sense, yes? It’s all pretty straightforward and intuitive? Not much more to say, right? If you are nodding your head non-sarcastically right now, then you don’t need my blog posts. Scram! For everyone else, this documentation has a lot of useful info and you will go back to it constantly, but it’s really more a reference than a guide and it doesn’t provide any good overview of what’s going on.

The next places to go are Apple’s CoreMIDI framework .h header files themselves, which like the documentation are very good for specific info about specific functions and types when you already mostly know what you’re doing. However, looking through random header files is generally not so great when you need an overall idea of what’s going on, want to decipher which files do what and how they work together, and need to just make some sort of sense of everything when you’re starting out and have no clue. So, keep moving…

Having some sample code to play with is good for learning things, but Apple has not yet provided any, so you’re out of luck again. You will instead want to check out the code that Pete Goodliffe mercifully provided on his blog with his PGMidi CoreMIDI sample project. Also updates here , here and here. Be sure to read through the comments on those blog posts, there’s some good stuff about implementing and testing MIDI there. We’ll talk about this code and how to use and tweak it in more depth in Part 2.

Peter Johnson of one red dog, maker of the excellent Molten Drum Machine, has generously put up some code that handles MIDI networking connection setup. This is used to configure your MIDI networking setup so your app can connect to other apps running on other devices and computers. We’ll cover this in Part 4.

Finally, for online discussion you can find a few small discussions on the official Apple developer forums in the Graphics and Media > CoreAudio section, and you can search through the official coreaudio mailing list for older, mostly MacOS based CoreMIDI discussions. Note that this official mailing list has the CoreAudio Apple engineers and other professional audio devs discussing fairly in-depth topics with each other, and they assume (nay, demand) you have a certain amount of knowledge before participating. It’s not the place to start learning things from scratch with very basic questions, but it can definitely be useful to see what sorts of bugs are being reported, and later on get some insight into alternate approaches to solving more difficult problems.

LET’S GET THIS PARTY STARTED

Alright, let’s get started. The first thing to keep in mind is that if you’re not already familiar with CoreAudio/CoreMIDI, and your iOS dev experience is mostly centered around Cocoa, then there will be major parts of this CoreMIDI stuff that will not be as intuitive to learn as some of the other more commonly used iOS features. The networking parts will make more sense, they are the same familiar NSObjects you have grown to know and love. At the lower levels where you’re sending and receiving raw MIDI data, however, it’s not object oriented and requires a completely different programming mindset. You probably already figured that out when you were poking around the CoreMIDI Framework Documentation and saw several sections with scary functions, pointers and typedefs.

You like Cocoa? Too bad.

I know, I know, the picture above is a bit of an exaggeration. You don’t actually want to literally throw those Cocoa books in the trash. You will probably still want to keep them around while working with CoreMIDI so you have something to hurl at the wall instead of throwing your iPad.

There is a bright side to all this: if you already use CoreAudio for the audio parts of your music app, and you probably do, then you should be familiar with a lot of similar concepts and approaches you’ll be using here. Glancing through the MIDI Services section of the framework documentation, you’ll see old friends like OSStatus, GetProperties, SetProperties, and all those lovely toll-free CFStringRefs. Of course on the not so bright side, we both know when I mentioned getting and setting CoreAudio properties you immediately put your head in your hands and started weeping uncontrollably, sobbing “not again… not again…” It’s okay, we’ve all been there.

So if this is all such a pain in the ass, why bother? Well, much like CoreAudio, yes, it can be frustrating to work with at times (it’s really not that bad, I’m mostly kidding around… mostly). But it’s also powerful, it’s been tested and refined for years, it’s part of the OS so you know it will be updated and will continue to work, and it’s better at what it does than anything else that’s available. And the end results, once you finally get it all working and assuming you have retained most of your sanity, are pretty cool.

Here’s what you get from CoreMIDI:

• Ability to communicate with just about all “real” external MIDI hardware
• Ability to communicate with other apps multi-tasking on the same device
• Ability to connect and communicate with apps on other iOS devices
• Ability to connect and communicate with a computer
• All of the above more or less work the same way, using the same API
• You get things like latency compensation not quite for free, but for very cheap
• MIDI may be old, and not perfect, but it works and everybody and everything uses it

More specifically, you get stuff like this:

That’s an iPad, two iPod touches, a Korg Electribe EMX-1 and Ableton Live all synced up and playing nice together. Pretty cool, and demonstrates how iOS music apps can fit right into a music studio beyond just being inspirational sketchpads and toys (which are also great, of course).

The video also shows the range of devices that MIDI lets you communicate with. It is a great example of the network effect: even if MIDI isn’t the most thoroughly specced protocol out there from a technical standpoint, it’s been around and used for so long and by so many things that the reality is it’s much more useful than anything else for the majority of musician users. This will likely also end up being true for CoreMIDI: the more apps that support MIDI in general and CoreMIDI specifically, the more they can interact with each other in cool ways and the more powerful and fun these music apps become for everybody. You don’t need to handle things in a special way for each and every app and setup you interact with, once you’ve figured out CoreMIDI your app can talk to everybody.

That’s the main reason I’m writing this series of posts. The more of us that figure this out and start using it, the better it gets for everyone. But most importantly, the better it gets for me. And really that should be your primary motivation here: making things better for me. Thanks!

LET’S GET DOWN TO BRASS TACKS. HOW MUCH FOR THE APP?

Korg R3 hooked up directly via its USB port and a Camera Connection Kit to an iPad

Okay, if you haven’t already, right now you need to grab some CoreMIDI apps and play around with them. If you’ve already done that, you need to do it a little more seriously and methodically. You need to get an idea of what these apps are capable of, what you might want your app to do, and most importantly you want to get familiar with some other apps so you have something to test with/against. Currently there are four major ways you will want to be using/testing CoreMIDI on iOS. They are, roughly in the order of complexity they will take to implement, from easy to harder:

• An iPad connected to CCK/USB, attached to a hardware synth
• An iOS device talking to a computer via WiFi
• Two iOS devices, talking to each other via WiFi
• Two apps talking to each other on a multi-tasking capable iOS device

The first iPad-CCK-USB scenario is the easiest both to set up and implement. A lot of apps that currently support CoreMIDI do this and only this. It requires an iPad (note this will change if/when hardware CoreMIDI interfaces for the iPhone/Touch appear) with a Camera Connection Kit and a USB MIDI Interface, along with a MIDI keyboard or controller of some sort. Some keyboards, like my Korg R3, have a built-in USB port with MIDI support that eliminates the need for a separate interface, so you just go iPad->CCK->keyboard.

iOSmidi.com has a great, comprehensive, updated list of MIDI interfaces that work with the iPad and the Camera Connection Kit. If you don’t already have a CCK and compatible interface for your iPad, go order them now. If you don’t have a cheap MIDI controller laying around, order one of those too. You need to have access to these sorts of USB MIDI configurations as you develop and test your app, because this is one of the most commonly used and requested MIDI setups.

Next, you need to stock up on CoreMIDI apps to experiment and test with. I recommend grabbing GarageBand to test keyboard MIDI input. Anything from Apple is an iOS de facto standard and this is the most popular music app out there, so you will absolutely want to make sure your app works well with it. You can also grab our free Little MIDI Machine app to test MIDI output. Yeah I’m plugging my own app, but hey it’s free. Also grab MIDI Monitor from iOSmidi.com to debug everything. There will be times when nothing is working correctly and you will want to use MIDI Monitor to figure out where things are going wrong. Hook everything up and get these apps working before you start trying to get your own app to work. You need an already working MIDI setup so if there’s a problem you know it is your code, and not anything else.

For the other three scenarios involving network and multi-tasking MIDI setups, I recommend buying two apps from Finger: BassLine and MoDrum. These two apps have an excellent MIDI implementation that works great with the iOS-computer setup, and they were the first two apps to implement the iOS device-iOS device and multi-tasking scenarios. They work perfectly, have a great MIDI setup page, and are excellent for testing with your own app. Get them, you will need them.

You should also grab Molten Drum Machine from one red dog to test these network setups. Peter Johnson has released his MIDI networking connection setup code which we’ll look at in Part 4, and you’ll likely refer to this code when you implement these connection setup features in your own app. It’s also just good to have a few different apps that do similar things, from different developers, so you can look at how they each do things differently, what one does better than others, and how well they work together.

And finally you’ll want a few Mac applications to test your iOS app connected up with a MacOS computer. You’ll need a sequencer like Ableton Live or Logic to interact with your app. You’ll want to move the Audio MIDI Setup utility into your dock if you haven’t already. And for testing and debugging I recommend MIDI Monitor for MacOS from snoize.com, which let’s you see incoming packets and their timestamps.

Using BassLine and MoDrum, along with GarageBand, MIDI Monitor, and Little MIDI Machine, create working setups for each of the other three non-USB CoreMIDI scenarios: iOS-computer, iOS-iOS, and multi-tasking. Figure out all the settings that are needed to connect various devices together, and play around with the multi-tasking scenario enough to see how awesome it is. As I mentioned before, the MIDI implementation of BassLine and MoDrum are really good, so everything should work. If it’s not working, sort out your hardware/setup problems using these apps, not with your own half-finished app.

Specifically, you should verify that you can get the following setups (or similar ones) working:

• MIDI keyboard plugged into USB MIDI Interface plugged into Camera Connection Kit plugged into iPad running Garageband. Play notes in Garageband using the keyboard.
• Little MIDI Machine on the iPad, plugged into the Camera Connection Kit, plugged into a USB MIDI Interface, plugged into a synth or sound module. Play notes on the synth with LMM.
• Ableton Live or another sequencer on a Mac, connected via WiFi network MIDI to an iPod/iPhone running MoDrum which is receiving MIDI clock from Ableton
• MoDrum running on an iPod/iPhone, connected via WiFi network MIDI to a Mac running Ableton Live which is receiving MIDI clock from MoDrum
• Little MIDI Machine running on an iPad, connected via WiFi network MIDI to a Mac running Ableton Live which is receiving MIDI notes from LMM which play a softsynth, while Ableton sends MIDI clock back to LMM and syncs it
• MoDrum on an iPod/iPhone, connected via WiFi Network MIDI to an iPad running BassLine which is receiving MIDI clock from MoDrum.
• Little MIDI Machine on an iPod/iPhone, connected via WiFi Network MIDI to an iPad running Garageband which is receiving MIDI notes from LMM.
• MoDrum and BassLine running simultaneously on the same iPad, synced up via virtual MIDI ports.

Multiple types of devices, multiple types of connection methods, multiple types of communication (notes, sync, etc)… and don’t forget multiple versions of iOS. CoreMIDI requires a lot of testing, and requires a lot of user documentation for your customers to be able to use it, on top of all the implementation work. Like I said before, it ain’t easy…

AND ONE MORE THING…

Before we wrap up Part 1, here are some random observations and quirks that might save you some time if you read about them here first. Most of the credit for these hints goes to Finger and his apps, I learned a lot about syncing and network connections just by looking at his MIDI Setup menu. You will too as you set up his apps and play with them. Sit down and spend some time studying his setup menu and figure out why he implemented each of the parts the way he did, it’s worth it. Here are a few of the little things I noticed:

Network latency setting in MoDrum

In MoDrum, note how the “Clock Delay” value increases from 0 to 50ms when you enable Network clock out. This is related to sending MIDI packets early to compensate for network latency, something we’ll talk about in Part 3. You’ll want to think in general about how your app will handle sending notes early to deal with latency on the send side for notes and clock, and on the receive side think about how you will handle sometimes being given clock messages a little late even if they have a correct timestamp.

Network connections setup in MoDrum

Next in the setup menu, look at the Network Connections list at the bottom of the setup screen. It will say “Searching…” if it can’t find anything to connect to, otherwise it will list possible connections (“iPhone 4G”, “Art’s MacBook”, etc). We’ll talk more about these in Part 4, but for now notice that devices automatically appear in this list whenever you load another app running a CoreMIDI Network Session on a device on the same network. If you quit that other app, the device will disappear from the list. You can make your Mac appear by loading Audio MIDI Setup. If you run the same MoDrum app on two separate devices, you’ll note that when you use the first device to connect to the second, the second automatically becomes connected to the first. These connections are two way, if you connect them on one end they are also automatically connected on the other.

Network Session setup in MoDrum

Again in the setup menu, the Network Session switch turns the CoreMIDI Network Session on and off, as you would expect. If you multi-task both MoDrum and BassLine on the same device, you’ll notice something more: if you turn off the Network Session in one, then switch to the other app, it will also be turned off there. This setting is device wide, and shared by all apps currently using CoreMIDI. This is something to keep in mind if you are enabling or disabling it without actively consulting the user, such as reloading a MIDI configuration from a saved preference file.

BassLine and FunkBox virtual port sources as seen by MoDrum

Our last observation about the MoDrum MIDI Setup menu involves multi-tasking and virtual ports. The screenshot above shows MoDrum running alongside BassLine and FunkBox on a single iPad. As you can see, BassLine and FunkBox offer virtual port sources so that they can send MIDI information to other apps. They will also have virtual port destinations listed in that section of the setup menu so they can receive data.

An important thing to note here is these ports are not app-app connections, these are direct ports to and from that specific app. Any other app can send to or receive from that same port. So if in MoDrum you turn on MIDI clock transmit to the BassLine virtual destination, you do not need to go to BassLine and turn on MIDI clock receive from MoDrum’s virtual source port. Those are two separate ports: one is always going into BassLine and anyone can send to it, one is always coming from MoDrum and anyone can listen to it.

This is different from the device-device and device-computer network MIDI connection list up above: with those, when you make a connection on one end, you automatically make a connection on the other end. Even if other apps don’t support setting up these connections, you can do it yourself from your own app and communicate with them. As long as one side is able to make the connection, it will work for both apps. For example, Little MIDI Machine on one iOS device can connect up with and send notes to Garageband or SynthX on another device, because Little MIDI can make those network MIDI connections even if the other apps can’t. After the connection is made, both sides can talk with each other.

With virtual ports, the connections are one way. Your app can send to FunkBox’s input port, and FunkBox can listen to your app’s output port, but if you configure them to do both FunkBox will receive every MIDI packet twice, once on its input and once on your output.

Enable Bluetooth, disable WiFi, Network MIDI still works

Moving on, another neat quirky thing I found when playing with network CoreMIDI is that it doesn’t have to be done over WiFi. If you turn off WiFi entirely and enable Bluetooth in the Settings app, you’ll note that paired devices can still discover, connect and communicate normally with each other using network MIDI over a Bluetooth connection.

This is cool for users who want a wireless MIDI setup that is not dependent on having access to an existing WiFi network. It’s cool for developers because it demonstrates just how powerful CoreMIDI is: it abstracts away those network layers for you, and as a developer for the most part you do not need to worry about whether these connections are being made over WiFi, USB or Bluetooth (this isn’t completely true because virtual ports work a little differently in some ways that we’ll cover later). In fact, both the Finger dev and myself were surprised when CoreMIDI over Bluetooth actually worked with our apps, we had done nothing to make it work and hadn’t even thought about it when writing the code.

Multi-tasking apps hang around in the background until manually stopped

Lastly, as you’re playing with the Finger apps synced up on a single device, note that since they hang around in the background when you switch out of them, there’s no easy way to automatically quit them. MoDrum and BassLine will be running in the background, even after the user leaves them to do something else, until they are explicitly cancelled by the user by holding down on the icon in the task bar and pressing the minus in the red circle. We’ll talk more about how you can minimize the effect of this on the processor and battery in Part 5.

USEFUL LINKS!

Here are some other sources to check out for info on CoreMIDI:

Apple’s CoreMIDI Reference

Pete Goodliffe’s PGMIDI Sample Code

Peter Johnson of one red dog’s CoreMIDI Network Setup Code

Michael Aldridge’s Sample Code to use CoreMIDI with Open Frameworks

UNTIL NEXT TIME…

Okay, that’s about it for Part 1. You have a working MIDI setup now with different apps and devices, so find some CoreMIDI quirks of your own, and start thinking about how your own app will be integrated into all these various setups. That will of course depend on the type of MIDI app you’re creating: are you a sound source to be played? A sequencer that sends notes? A self contained app that needs to sync up in time with other things? Think about how you can test your app so it works in these various scenarios, and if possible find another app from someone else that does similar things and fills a similar role, so that when you run into problems you can doublecheck if it’s just your app that isn’t working, or if it doesn’t work with anybody’s apps.

Now that you’ve played with CoreMIDI as a user and you’re sufficiently comfortable using it, you’re ready to add some basic MIDI functions to your own app, which we’ll cover next in Part Two.

Little MIDI v2 for iPad

Little MIDI Machine got littler! Now for iPhone and iPod Touch too

Version 2.0 features include:
Now a universal app, supports iPad, iPhone, iPod Touch
CoreMIDI support, including network MIDI
MIDI clock sync for both Line 6 and CoreMIDI, send and receive
New random sequence mode
Updated graphics/UI
Still free

And more! Coming soon!

While we wait, if you’ve been enjoying the latest FunkBox updates and get a chance to leave an app store review (or update an old one), we’d really appreciate it! Every time we release an update it hides all the old reviews, which actually hurts us and has a noticeable negative effect on sales. So it really helps us out and encourages us to release more updates if you leave new reviews or update the old ones so they re-appear again. Thanks so much!

Help us out and leave us a FunkBox review at the iTunes App Store