Multi-packet SysEx commands should be coalesced

[snej]

I'm working on receiving patch/configuration dumps from the Novation Circuit and Novation Launch Control XL. These are sent by the device as SysEx messages that are about 300 bytes long. Each such message is received as multiple MIDI packets.

MIKMIDI doesn't handle these messages correctly. It's interpreting each packet as a separate command, instead of part of one long command. For example, here's the list of commands I receive when the Launch Control XL sends its configuration:

<MIKMIDISystemExclusiveCommand: 0x61000004ca80> time: 10:10:39.159 command: 240 universal: 0 sysexChannel: 0 
	data: <f0002029 02117700 0d00007f 00000e00 007f0000 0f00007f 00001000 007f0000 1100007f 00001200 007f0000 1300007f f7>
<MIKMIDICommand: 0x6000000282c0> time: 10:10:39.156 command: 0 
	data: <00001400 007f0000 1d00007f 00001e00 007f0000 1f00007f 00002000 007f0000 2100007f 00002200 007f0000 2300007f>
<MIKMIDICommand: 0x600000027e60> time: 10:10:39.157 command: 0 
	data: <00002400 007f0000 3100007f 00003200 007f0000 3300007f 00003400 007f0000 3500007f 00003600 007f0000 3700007f>
<MIKMIDICommand: 0x6000000282e0> time: 10:10:39.157 command: 0 
	data: <00003800 007f0000 4d00007f 00004e00 007f0000 4f00007f 00005000 007f0000 5100007f 00005200 007f0000 5300007f>
<MIKMIDICommand: 0x600000028320> time: 10:10:39.157 command: 0 
	data: <00005400 007f0000 0029007f 0000002a 007f0000 002b007f 0000002c 007f0000 0039007f 0000003a 007f0000 003b007f>
<MIKMIDICommand: 0x608000031080> time: 10:10:39.157 command: 0 
	data: <0000003c 007f0000 0049007f 0000004a 007f0000 004b007f 0000004c 007f0000 0059007f 0000005a 007f0000 005b007f>
<MIKMIDICommand: 0x608000031120> time: 10:10:39.157 command: 0 
	data: <0000005c 007f0000 0069007f 0000006a 007f0000 006b007f 0000006c 007f0000 0168007f 00000169 007f0000 016a007f>
<MIKMIDICommand: 0x6080000310e0> time: 10:10:39.158 command: 0 
	data: <0000016b 007f0000 0d0f001d 0f002d3e 003d3e00 4d3c005d 3c006d2f 007d2f00 0e0f001e 0f002e3e 003e3e00 4e3c005e>
<MIKMIDICommand: 0x6080000310a0> time: 10:10:39.158 command: 60 
	data: <3c006e2f 007e2f00 0f0f001f 0f002f3e 003f3e00 4f3c005f 3c006f2f 007f2f00 f7>

This took me some time to figure out. What's going on is:

• As per spec, a SysEx message ends with a 0xF7 byte.
• The 0xF7 at the end of the MIKMIDISystemExclusiveCommand's data is not real: the class's -data method appended it because it didn't see a real 0xF7 at the end of the packet data.
• The rest of the commands aren't commands at all, which explains why they have weird types like 0x00 and 0x3c. They're just the continuation of the SysEx data.
• The last 'command' ends with the real 0xF7 byte, signaling the real end of the SysEx command.

What MIKMIDI should be doing when it receives a SysEx command that doesn't end with 0xF7 is to hold the command; then each successive packet received should be appended to the SysEx's data, until one of them ends with 0xF7. Then the SysEx command is complete and can be dispatched to the app.

If it did so, the correct SysEx command object would look like this:

<MIKMIDISystemExclusiveCommand: 0x61000004ca80> time: 10:10:39.159 command: 240 universal: 0 sysexChannel: 0 
	data: <f0002029 02117700 0d00007f 00000e00 007f0000 0f00007f 00001000 007f0000 1100007f 00001200 007f0000 1300007f
	00001400 007f0000 1d00007f 00001e00 007f0000 1f00007f 00002000 007f0000 2100007f 00002200 007f0000 2300007f>
	00002400 007f0000 3100007f 00003200 007f0000 3300007f 00003400 007f0000 3500007f 00003600 007f0000 3700007f>
	00003800 007f0000 4d00007f 00004e00 007f0000 4f00007f 00005000 007f0000 5100007f 00005200 007f0000 5300007f>
	00005400 007f0000 0029007f 0000002a 007f0000 002b007f 0000002c 007f0000 0039007f 0000003a 007f0000 003b007f>
	0000003c 007f0000 0049007f 0000004a 007f0000 004b007f 0000004c 007f0000 0059007f 0000005a 007f0000 005b007f>
	0000005c 007f0000 0069007f 0000006a 007f0000 006b007f 0000006c 007f0000 0168007f 00000169 007f0000 016a007f>
	0000016b 007f0000 0d0f001d 0f002d3e 003d3e00 4d3c005d 3c006d2f 007d2f00 0e0f001e 0f002e3e 003e3e00 4e3c005e>
	3c006e2f 007e2f00 0f0f001f 0f002f3e 003f3e00 4f3c005f 3c006f2f 007f2f00 f7>

I've added logic like the above to my app, and it receives the dumps correctly now. I had to access the private internalData property of the MIKMIDISystemExclusiveCommand in order to get its true data without the phantom trailing 0xF7, though.

I took a look at adding the same logic to MIKMIDI's MIKMIDIPortReadCallback function, but there's already some complex stuff going on there with buffering 14-bit CCs, involving dispatch queues, so I didn't feel confident extending that code. But it probably wouldn't be difficult for someone already familiar with this part of the codebase.

[ghost]

I submitted a pull request about a year ago here:
#130

I didn't really know what I was doing at the time but this seemed to do the trick. I was trying to handle Roland DX50 et al SysEx.

[armadsen]

Fixed by #200.