Acknowledgement packets decoded

[ammaraskar]

Hey folks, thanks for the excellent project. I spent a few days reverse-engineering the base station's firmware to try to figure out how ACK packets were done. The full nitty gritty details are here https://blog.ammaraskar.com/iclicker-reverse-engineering/

but here's the cliff notes and an implementation for iSkipper to respond with acks for answer packets:

#define ACK_HEADER_SIZE 3
#define ACK_SIZE 5
const uint8_t ack_header[ACK_HEADER_SIZE] = {0x55, 0x55, 0x55};

void iClickerEmulator::sendAck(iClickerAnswerPacket_t* answer, bool accept) {
    uint8_t ack_payload[ACK_SIZE];
    encodeId(answer->id, ack_payload);

    ack_payload[2] = ~ack_payload[2];
    if (accept) {
        ack_payload[3] = (ack_payload[3] & 0xF0) | 0x2;
        ack_payload[4] = 0x22;
    } else {
        ack_payload[3] = (ack_payload[3] & 0xF0) | 0x6;
        ack_payload[4] = 0x66;
    }

    _radio.setSyncAddr(ack_header, ACK_HEADER_SIZE);

    _radio.setFrequency(_radio._chan.recv);
    _radio.setPayloadLength(ACK_SIZE, false);

    _radio.send(ack_payload, ACK_SIZE, false);
}

I was testing this by inserting the line _self->sendAck(&recvd.packet.answerPacket, false); into isrRecvCallback

Essentially, the acknowledgements echo back the encoded id but they change the 3rd and 4th bytes. Namely, the 3rd byte is bitwise inverted and the 4th byte (with the answer nibble) has a magic value ORd with it. This is followed by either 0x22 to send a tick to the iClicker or 0x66 to send the closed message.

[orangeturtle739]

Whoa! That's pretty cool, and the only missing part!

[wizard97]

Wow fantastic work! The idea of looking at the firmware update's to get a hex dump was very cool. We have been wanting for someone to reverse engineer this base station for the the last couple of years.

Would you be willing to submit a pull-request with your changes so that we can incorporate this?

[wizard97]

Sorry for the delay in merging this in. I had a chance to read your detailed blog post, and the acknowledgement protocol seems to disagree with what we discovered. While our reverse engineering focused on the answer submission protocol, we still discovered aspects about the acknowledgement.

Would you mind taking a look at our report (linked in the README)? In our SPI dumps, we find the remote listening for a 7-byte payload (not 5 byte) with a sync address generated from the encoded ID rather than the fixed one you gave. Any ideas where this disagreement comes from? How have you tested that your acknowledgement works?

[ammaraskar]

Any ideas where this disagreement comes from?

If I read the report correctly, it says that the ack packet should have a 2 byte sync addr consisting of the encoded ID and then 7 bytes of payload. However, I can't find anywhere in the base station's firmware where it ever writes out 9 bytes.

The valid values are (all including the preamble): 4, 7, 8, 11, 12, 19

Also, I'm not sure if it makes much sense to have the radio sync addr for the ACKs be based off the encoded id. The base station sends acknowledgements sequentially on one thread, thus each remote needs to be able to check the ACKs being broadcasted and see if it matches theirs. They don't know the IDs for the other remotes, nor do they listen for answer packets so they can't be changing their internal sync addr.

If I recall correctly from testing, if you use the wrong sync addr, you essentially end up with noise which might be why your captured ACKs look so random.

How have you tested that your acknowledgement works?

Yup, the code linked in the PR successfully manages to get my iClicker2 to show both the tick and the closed messages on screen.

[wizard97]

If I read the report correctly, it says that the ack packet should have a 2 byte sync addr consisting of the encoded ID and then 7 bytes of payload. However, I can't find anywhere in the base station's firmware where it ever writes out 9 bytes

Keep in mind, the way you set the sync address is completely different from how you actually write the payload. The payload length field that you set on the radio does not include the sync address size or the preamble length.

Also, I'm not sure if it makes much sense to have the radio sync addr for the ACKs be based off the encoded id. The base station sends acknowledgements sequentially on one thread, thus each remote needs to be able to check the ACKs being broadcasted and see if it matches theirs. They don't know the IDs for the other remotes, nor do they listen for answer packets so they can't be changing their internal sync addr.

I'm confused what you mean by this. What you describe is the exact behavior that one would want. An individual remote only wants the acknowledgement for it's submission. If the sync address was not based off the encoded ID and rather fixed like you describe, the remotes would have to filter through all the acknowledgements going to all the remotes in software. This not only wastes the remotes battery power, but it could potentially miss it's own acknowledgement if it can't process them fast enough.

If I recall correctly from testing, if you use the wrong sync addr, you essentially end up with noise which might be why your captured ACKs look so random.

Yeah this is correct and has to do with the sensitivity setting of the reciever. If you increase the threshold, it won't pick trigger on background noise.

Yup, the code linked in the PR successfully manages to get my iClicker2 to show both the tick and the closed messages on screen

Awesome, this was great work! I just want to resolve the inconsistency between our SPI dumps, or at least figure out an explanation for it before I merge this in.

One thing you should keep in mind is it should be equivalent if the base station transmits the sync address by manually prepending it to the payload. If you account for this, then your reverse engineering actually almost agrees with what we found. If you look at your blog post, the sync address (first two encoded bytes of remote ID) we see the remote setting over SPI, is bytes 3 and 4 of what you see being transmitted in the base station firmware. Perhaps the remote's sync address functionality is triggering on these bytes? This still does not explain the 7 byte payload we observe though, but it's possible the trailing four bytes that the remote captures are just ignored (since they will be garbage from the background noise )

[ammaraskar]

The payload length field that you set on the radio does not include the sync address size or the preamble length.

The firmware actually doesn't seem to use the radio's payload length register at all when writing. It places the entire radio payload (including the sync bytes) in memory and essentially uses a for loop with the number of bytes to write.

it won't pick trigger on background noise

I was thinking of a different issue, though this is just speculation on my part. I am no expert in radio transceivers. Let's say the sync addr from your encoded ID ends up being: 0x14 0x8C = 00010100 10001100

and for mine the full payload ends up being:
0x5B 0x78 0xD6 0x72 0x22 = 01011011 01111000 11010110 01110010 00100010

If we overlay these two together, you can see how similar the pattern is, and if the transceiver mistakes the middle of my payload as your sync bytes then it would end up reading garbage.

0101101101111000110101100111001000100010
             0001010010001100

In contrast, the other patterns I see used in the firmware like 0x55 0x55 0x55 = 01010101 01010101 01010101 or 0x36 0x36 0x36 = 00110110 00110110 00110110 seem like they are long enough and have a nice high-low pattern to ease detection.

Perhaps the remote's sync address functionality is triggering on these bytes?

That's entirely possible, you have a good point about the battery concern, I haven't looked at the remote's firmware yet but if it puts the AVR into sleep mode and uses interrupts from the radio chip then that would certainly make sense.

[wizard97]

he firmware actually doesn't seem to use the radio's payload length register at all when writing. It places the entire radio payload (including the sync bytes) in memory and essentially uses a for loop with the number of bytes to write.

Interesting, it looks like the base station just doesn't use the sync address functionality of the chip then, and rather just manually prepends them to the payload.

If we overlay these two together, you can see how similar the pattern is, and if the transceiver mistakes the middle of my payload as your sync bytes then it would end up reading garbage.

I see the point you are trying to make, but if the SNR was that low such that a bit (or several) were incorrect, you would have a very high likelihood of having a bit (or several) flipped in the actual payload as well and would receive an incorrect message.

Based on the SPI dumps and your reverse engineering efforts, I'm pretty sure the remote must be triggering on this 3rd and 4th byte received. I'm going to play around with this and see if I'm right.

[ammaraskar]

it looks like the base station just doesn't use the sync address functionality of the chip then

yup, seems like it only uses it for receiving packets, not sending them.

must be triggering on this 3rd and 4th byte received

do you have any insight on why the base station sends the 0x55 0x55 0x55 premable? Or why the remote doesn't use the full 0x55 0x55 0x55 byte_3 byte_4 as the sync addr?

[wizard97]

do you have any insight on why the base station sends the 0x55 0x55 0x55 premable? Or why the remote doesn't use the full 0x55 0x55 0x55 byte_3 byte_4 as the sync addr?

So remember 0x55 is 0b01010101 (notice how the bits are alternating). The point of a preamble is to help the receiver learn the channel. The receiver is constantly trying to determine the decision boundary for a 1 and 0. By sending this alternating pattern for a while, the receiver figures out what should be considered a 1 and what should be considered a 0, as it tries to set the boundary such that half the bits are classified as a 1 and the other half as a 0. Thus, 0x55 repeated for a while is a particularly good pattern for the receiver to learn the channel and accurately trigger on the sync address that follows after the preamble.

[wizard97]

Hi,

Sorry again for the delay. I had a chance to test out your implementation on the iClicker1, the iClicker+, and the iClicker2. On the iClicker2 testing with your updated capture sketch I only see the remote successfully getting the ack ~75% of the time, the other times it reports no base station.

I also tested this on the iClicker1 and the iClicker+, it unfortunately does not seem to work at all. Did you have a chance to test on these remotes? Any ideas what might be wrong? I wonder if it could be some sort of timing issue?

Edit:
The issue causing the 75% success rate was the following piece of code being in the wrong place:

  // restore the frequency back to AA and go back 
  // to promiscous mode if we sent an ACK packet
  if (SEND_ACKS) {
    clicker.setChannel(iClickerChannels::AA);
    clicker.startPromiscuous(CHANNEL_SEND, recvPacketHandler);
  }

You were running this every void loop() iteration. In the worst case this can create a 100ms delay from the time an ack is sent to the time it takes the radio to go back into promiscuous mode. Moving this inside the recvPacketHandler() ISR fixes this problem and the iClicker2 appears to work 100%. However, the iClicker1 and the iClicker+ still do not work. It looks like you there is still a piece missing in the protocol for the ack.

[ammaraskar]

the iClicker1 and the iClicker+ still do not work

Aah, I don't have one of those devices to test, but it's interesting that it's not backwards compatible, will take another look when I get some time and report back.

[ammaraskar]

Since I won't be able to get the older remotes for a few days, I was thinking of looking at the remote's firmware, you guys mentioned you got the ID obfuscation scheme from https://github.com/zmcginty/iclicker/blob/master/iClicker%20flash%20unfuckedwith.asm ?

Do you know if there's a .hex version of it somewhere?

[wizard97]

the iClicker1 and the iClicker+ still do not work

Aah, I don't have one of those devices to test, but it's interesting that it's not backwards compatible, will take another look when I get some time and report back.

Yeah, I was just as surprised about this as you... One potential thing we never considered is it is possible there is some meta-data encoded in the answer submission that describes the type of remote. The base station could then use this to determine how to form the ack. I'm not sure though...

I don't have a base station, but I think if we were able to get captures of the ack across different remotes, we could figure this out real quick. See #13

[wizard97]

Since I won't be able to get the older remotes for a few days, I was thinking of looking at the remote's firmware, you guys mentioned you got the ID obfuscation scheme from https://github.com/zmcginty/iclicker/blob/master/iClicker%20flash%20unfuckedwith.asm ?

Do you know if there's a .hex version of it somewhere?

Try these. Not sure if they are from the base station or remote:
https://github.com/zmcginty/iclicker
https://github.com/mcpherrinm/iclicker

[charlescao460]

@ammaraskar @wizard97
First, I'd like to thank you for your brilliant reverse-engineering works!

So, I figured out how to capture ACKs from a real I>Clicker 2 base.
I am using my own iClicker 2, ID is {0x96,0x8C,0x71,0x6B},
and after encoding it becomes {0xB5 0x18 0x38 0x30}.
I set the sync address to be {0x55,0x55,0x55,encodedID[0],encodedID[1]}. Now we only need to verify the first 3 bytes, since the previous 5 bytes are verified by sync address.

Good news is your conclusion about "Closed" ACK is completely right. When I close the base, I got C7 36 66, which is consistent with your findings.

However, for "Accepted" case, the result is a little bit weird. After sending answer A, I got 38 31 XX, where XX looks like a random number. And 38 31 are actually payload[2] payload[3] of answer packet.
I didn't see any constant like 0x22.
Here are some sample of my captures: (They are 7-bytes because I am still using iSkipper's library, you can see my codes in #13 ).

C7 36 66 55 B6 84 54 
38 31 FC 93 7C CD 47 
38 31 DB D4 F3 68 5A 
38 31 E6 DF 17 BF F3 
38 31 D9 42 2D F8 A9 
38 31 EE 65 AC 9F 14 
38 31 CB 25 56 FF 31 
38 31 E2 BA BC 1C FE 
38 31 F9 1E BA 1D 9E 
38 31 BE 49 1F 2F 66 
38 31 FF F7 55 19 93 
38 31 B3 DC DE 50 57 
38 31 EC 29 3D B4 7D 
38 31 D5 8A DB BD 1 
38 31 FF AE CF 4E 9B 
38 31 F4 F9 9 30 89 
38 31 F7 F9 87 FD ED 
38 31 BF 5E E7 FB 2F 
38 31 AA 23 AE 5 3E 
38 31 DB 79 CB ED D9 
38 31 C6 99 CF 48 1C 
38 31 81 DB 2 5B A3 

Therefore, I changed your acknowledgeAnswer() to be:

void iClickerEmulator::acknowledgeAnswer(iClickerAnswerPacket* packet, bool accept) {
    uint8_t ack_payload[ACK_SIZE];
    encodeId(packet->id, ack_payload);
    if (accept) {
        ack_payload[3] = (ack_payload[3] & 0xF0) | (0x0F & getAnswerOffset(packet->answer)) ;
        ack_payload[4] = 0x22;
    } else {
        ack_payload[2] = ~ack_payload[2];
        ack_payload[3] = (ack_payload[3] & 0xF0) | 0x6;
        ack_payload[4] = 0x66;
    }

    _radio.setChannelType(CHANNEL_RECV);

    _radio.setSyncAddr(ACK_SYNC_ADDR, ACK_HEADER_SIZE);
    _radio.setPayloadLength(ACK_SIZE, false);

    _radio.send(ack_payload, ACK_SIZE, false);
}

And this ACK is successfully accepted by my I>Clicker 2 remote. @wizard97 could check for iClicker 1 and iClicker +. If we're lucky, we know all about ACKs!

(I am very curious why iClicker 2 accept these two different ACKs. If what I found is the old-version of ACKs, there is no reason for a iClicker 2 base sending a old ACK to a iClicker 2 remote. Maybe one day they'll update base firmware and force everyone to buy a iClciker 2 ? This is one resonable explanation.)

[ammaraskar]

And 38 31 are actually payload[2] payload[3] of answer packet.

There is a code path that literally makes the ack bytes what the original payload was, I assumed it was unused. In that branch the ACK length is 4 bytes, which is likely why you see a random byte XX, it's just noise.

Side question: would you be interested in code to send a welcome message and the other question modes since you're trying to emulate a full base station?

[charlescao460]

@ammaraskar Now we'll just wait @wizard97 to check the validity of ACKs, since I don't have iClicker 1 or iClicker+ neither. (I have a old iClicker 1 base, but currently I am only taking iClicker 2 base with me)

Side question: would you be interested in code to send a welcome message and the other question modes since you're trying to emulate a full base station?

Yes, I'll definitely want those codes and functions. I'll be able to capture real welcome message and compare to your findings. And I saw you mentioned on your blog that

Side note: I haven’t documented the ACK packets for other question modes since I figured there’s not a lot of interest for those, please let me know if you’d like to see those.

It will be great if you could make those type of questions available. Since some lectures in my University are using those type of questions to prevent students operating multiple iClicker in the same time. (The operations of numerical and alphabetical question are inhuman, therefore you can't using more than one iClicker at the same time for those type of questions.) It would be incomplete for our reverse-engineering if we couldn't make all type of questions usable.

I guess that numerical and alphabetical question may share the same packet format, since they have exactly the same length. "Multiple Numeric" and "Multiple Alphanumeric" are not shown on latest iClicker software. So... maybe it will be just a few more little works for you to do.

And if you are tired of implementing those finding by yourself, I can help you doing that.

[wizard97]

void iClickerEmulator::acknowledgeAnswer(iClickerAnswerPacket* packet, bool accept) {
    uint8_t ack_payload[ACK_SIZE];
    encodeId(packet->id, ack_payload);
    if (accept) {
        ack_payload[3] = (ack_payload[3] & 0xF0) | (0x0F & getAnswerOffset(packet->answer)) ;
        ack_payload[4] = 0x22;
    } else {
        ack_payload[2] = ~ack_payload[2];
        ack_payload[3] = (ack_payload[3] & 0xF0) | 0x6;
        ack_payload[4] = 0x66;
    }

    _radio.setChannelType(CHANNEL_RECV);

    _radio.setSyncAddr(ACK_SYNC_ADDR, ACK_HEADER_SIZE);
    _radio.setPayloadLength(ACK_SIZE, false);

    _radio.send(ack_payload, ACK_SIZE, false);
}

Wow, fantastic work. I can confirm your change fixed the iClicker+ and still works with the iClicker2. However, you're not going to believe this, but it still does not work with the original iClicker...

EDIT It does indeed work on the iClicker1! I did not set IS_RFM69HW to true, which caused it not to transmit with the PA on.

[wizard97]

Side question: would you be interested in code to send a welcome message and the other question modes since you're trying to emulate a full base station?

I personally would love for support of this to be incorporated into this library.

[charlescao460]

However, you're not going to believe this, but it still does not work with the original iClicker...

From iClicker website, iClicker + and iClicker 2 are current supported product, I guess they are using different protocol from iClicker 1.
Luckily I have both versions of bases. I'll capture the ACKs from my old iClicker base, sometime in this summer.

But for now, I believe we can just merge this function to master? In my university, it is hard to see someone using old iClicker 1. (The oldest I saw is a iClicker 2 with ID starting with '55', the latest iClicker 2 has ID starting with 'A5', I can't imagine how many iClicker has been sold)

[charlescao460]

EDIT It does indeed work on the iClicker1! I did not set IS_RFM69HW to true, which caused it not to transmit with the PA on.

Great! We finally successfully figured it out! 🙏

[ammaraskar]

Great work everyone, I'll amend the PR to use the shorter backward compatible payload and then make separate PRs for the other question modes/welcome message etc.

[wizard97]

Great work everyone, I'll amend the PR to use the shorter backward compatible payload and then make separate PRs for the other question modes/welcome message etc.

Fantastic, I will accept this right away! One request, could you use the configureRadio() method instead of:

    _radio.setChannelType(CHANNEL_RECV);
    _radio.setSyncAddr(ACK_SYNC_ADDR, ACK_HEADER_SIZE);
    _radio.setPayloadLength(ACK_SIZE, false);

and also get the withAck correctly working in submitAnswer() with the reverse engineering of the ack that was just done?

You will probably have to update some of the constants such as the poorly named macro RECV_SYNC_ADDR_LEN

[wizard97]

Since this is really becoming a cross university effort, I would like to add a section in the README to reflect the universities where students have contributed from.

We also need to update the README to reflect the current state of the reverse engineering to include the acks. I would also like to add the additional stuff that needs to be reversed engineered, such as the as the other question modes.

Would one of you be interested in updating the README?

[charlescao460]

Would one of you be interested in updating the README?

Yeah I'll be willing to doing that. @ammaraskar, could you please tell me about your university, or you prefer being listed as an individual contributor?

Next, I'll start to work with my iSkipper-Softwareto make these updates.

By the way, @wizard97, I saw you create a new repo OpenClick early in this year. And I'd like to update my iSkipper-In-One-Package to using a more powerful MCU. Would you be interested in developing this togther? I can easily order PCB and chips from China in a very low price.

[ammaraskar]

One request, could you use the configureRadio() method instead of ...

Will do.

get the withAck correctly working in submitAnswer() with the reverse engineering of the ack that was just done

sounds good, I will need someone with a base station to test those changes, so I'll make it a separate PR.

could you please tell me about your university

you can put me down with Purdue University.

[wizard97]

@wizard97, I saw you create a new repo OpenClick early in this year. And I'd like to update my iSkipper-In-One-Package to using a more powerful MCU. Would you be interested in developing this togther? I can easily order PCB and chips from China in a very low price.

I was making an open source remote replacement that included a PCB and a 3D printable case design, but I never quite finished it. This was then going to be used for some soldering projects that we host at Cornell. However, I just graduated last week, so I am no longer a student. I might still be interested in working on this project, but I will have to see how motivated I am in a couple months when I start working. Maybe ask me again then? Or do you want to work on it over the summer?

[charlescao460]

Congratulation on your graduation !
I'd like to do this over the summer. The idea that developing a open sources version is definitely great. I am not majoring in EE so designing circuits is a little bit difficult for me. I'll need some help on it.
Speaking of 3d printing, I am now doing internship in a 3D printing company, which allows all employee using all kinds of printer, so it would be very convenient for us to do development this summer.

[wizard97]

Congratulation on your graduation !
I'd like to do this over the summer. The idea that developing a open sources version is definitely great. I am not majoring in EE so designing circuits is a little bit difficult for me. I'll need some help on it.
Speaking of 3d printing, I am now doing internship in a 3D printing company, which allows all employee using all kinds of printer, so it would be very convenient for us to do development this summer.

Thank you! I could certainly do the schematic and PCB stuff, but again it comes down to my motivation :). Feel free to email me and we can chat more.