complement/help with similar project

[Lurkars]

Hello, I am working on a similar project esp-ena on GitHub for ESP32. Since we may run into similar issues, we can support each other. Since I am hobbyist in C, would also be nice to have a professional look at my code base.

[olafland]

Great, I had a quick look at you code base. You support even more features than we do in this version.

[pinkit-cwa]

Dear @olafland and @Lurkars,

I also want to contribute. I am fluent in C and "small devices" - I make my living by developing portable smart card operating systems.

I already had a look at Olafs work about a week ago, and will now check esp-ena. Maybe both projects could be merged, or at least share duplicate code?

I am currently working on some "middleend", to connect Corona-Warn-App to the bracelets. I hope that way, we can save a whole lot of front-end development. Please find attached a first version of some README.txt that I plan to use for this project, for now called "CovidMiddleend"

[pinkit-cwa]

I for now had a short glance at esp-ena. I agree with @olafland impression that this has more features (supports on-board RTC and display). But, I also see that it is limited to ESP32, while CovidBracelet is based on Zephyr Project, which abstracts the hardware / should "run everywhere".

We all seem to want to use different hardware:

• @olafland uses NRF52, obviously supported by Zephyr
• @Lurkars uses ESP32. Zephyr probably supports that too: https://docs.zephyrproject.org/latest/boards/xtensa/esp32/doc/index.html
• I would prefer some coin-cell based bluetooth-capable board like https://docs.zephyrproject.org/latest/boards/arm/efr32mg_sltb004a/doc/index.html

So I think it would be useful to port @Lurkars code to Zephyr. I can do that. I just need to learn Zephyr myself first.

[pinkit-cwa]

Also we might get into a license issue:

@Lurkars uses a copyleft license. I think specifically for Covid bracelets this choice makes a lot of sense. If you buy a device with ENF aka ENA integrated, you want to be able to verify the source code and possibly upgrade the firmware.

On the other hand, @olafland uses an Apache style license, which might encourage more parties to integrate the software into their hardware. I do have slight preference for Copyleft, but on the other hand I also do want this to be used by as many people as possible.

[olafland]

@pinkit-cwa sweet, any help is welcome. I had a quick look at your Readme and it looks good. I have some deadline in the next days, but will look at this again next week. Regarding the license. I think we are flexible. We choose Apache as its is the license used by Zephyr, our underlying OS. This makes it much easier when it comes to integration with the OS. Nonetheless, we care more about overalll progress than a specific license.

[Lurkars]

@pinkit-cwa thanks for your contribution so far. Here my thoughts about it:
First I think the approaches differ a bit (correct me if I am wrong): I want to create a full device without the need of any middle-ware. I am still not sure if this works out, because of the very limited memory, but I think with a relay server putting the Exposure Key exports into much smaller chunks it should be generally be possible. I think the bracelet approach will need some middle-ware because the capabilities are even more limited than the ESP32. The choose of ESP32 came to: it is cheap, has WiFi and BLE. Without WiFi a middle-ware will always be required.

On the Zephyr topic: ESP-IDF is based on FreeRTOS, so it is also a wide spread RTOS, quick search that NRF52 also supports FreeRTOS but I am not familiar with this, so maybe you can tell more yourself. I am not familiar with Zephyr, I think it abstracts more than just RTOs? So BLE and WiFi code could be shared with more devices on a switch to Zephyr? I can think of this, but will have to check how easy it will be for me to port.

The license: besides that I am a big fan of copyleft, the main reason for me is my own lack of knowledge in the C/embedded section, so that I can learn from contributions because of copyleft. I don't want that someone finalize a device with my code and I can just gaze at the end product/binaries. Also I think it is the best fit for DIY makers, that they can easily create own devices.

Form my side, I will make a big update on the weekend where I finalized big parts of the user interface. Afterwards I will start with the big problems, that due to memory limitations I am not able to unzip an Exposure Key export for now. I still appreciate tips for my code like "do"s and "don't"s and most important memory optimizations because as said I am a hobbyist in that section and for sure have some "trial and error" parts in my code. Update will come on 15/16. August.

[pinkit-cwa]

@olafland @Lurkars

tldr - just stuff about my background and motivation

Well, I neither am experienced in Zephyr nor FreeRTOS nor any other RTOS. I am quick in reading and understanding alien code and documentation, and interested in about everything bare-bone, best at the expert level.

I am an expert in secure fast portable native smart card operating systems, and earn my money there (ISO7816-4, mostly RFID ePassports, but we also are currently working on a Javacard).

I like to extend my knowledge / prepare for the future / not just work on the current job ("Freigeist" I was called). Also, our company (which I partially own) considers to extend towards IOT (providing security for IOT - "the 's' in IOT stands for security"). So I am sure this project is exactly right for me. Also, in my company there are at least 3 out of 10 people who would like to do CWA, but don't have the appropriate means. I for example don't carry a smartphone. I do have some old ones that cannot send beacons, just receive them. Also

I already prepared for this some weeks ago (between working on other stuff, downloading packages for later checking). When I discussed my off-job activities with my boss two weeks ago, he said would also like to have some specific ENF hardware. Doing off-job stuff is quite common in our company.
A week ago I got an aching throat. Diagnosis was "Mandelabszess". I could not drink anymore, so I went to the hospital (no good to wait until you are dehydrated, so if you cannot drink, go to the doctor or hospital). They resected, and I remained in the hospital until today. There I was isolated (due to people with covid in contact with other people in contact with me), so I had a lot of time to spend in front of my laptop.
And this week I am again in some kind of isolation, as my family is going to enjoy out long-planned and payed trip to Italy, but I cannot go on holidays with them due to "erhöhtes Blutungsrisiko". I am still on sick-leave, so I cannot really work for my company (as I only can work within the secured environment at our company).

Last week (amongst other), I:

• checked my downloaded "interesting stuff", and e.g. deleted some of that (e.g. IOsonata after I found that Zephyr seems to be a more elaborated HAL)
• set up another VM for Ubuntu 20.04 (been there done that)
• first time installed Android Studio IDE (IntelliJ IDEA), and started debugging my (very slightly modified) version of CWA, en-passant learning kotlin and current features of JavaScript (Google Mobile Services "Task" concept) etc.
• put together my notes
• talked to the personell at the hospital, because I noticed almost no ENF beacons flying around. Turns out they must not use their smartphone during work. Also a bracelet would not be accepted. "Fussfessel" or something to put into your pocket would be an option.

[pinkit-cwa]

I also care more about progress in this project than other things

@olafland, is my impression correct that you are an expert in Zephyr OS? Because if so, I would like to use your expertise:

1. Can you please confirm my first impression that Zephyr is not just an RTOS (which is "scheduling"), but indeed (as suggested also by @Lurkars) includes a general purpose HAL, i.e. Hardware Abstraction Layer that can be used to access various hardware resources on various platforms, I.e. "port once, run anywhere"?
2. According to ENF specs, if possible, the BLE MACs shall be randomized, and this shall happen in sync with the changing of the sent exposure notification beacons. I did not see tat in CovidBeacon (please confirm), but it is contained in ESP-ENA. In this context, can you quickly point me to some function of Zephyr that can be use to change the BLE MAC address? If not I'll find out myself, but if you can quickly answer this question, then I save some time.

My next steps are:

• build and (if possible without hardware) debug CovidBracelet
  • set up development environment
  • download all base packages
  • whatever else that is missing here
• shop some NRF, ESP and ERF devices

I already tried by just issuing 'cmake .', but that failed. Next I will try to follow ZephyrProject documentation on how to build some of their sample projects, then see if I can apply what I learned to compile CovidBracelet.

@Lurkars

Once that works I will continue by porting ESP-ENA to Zephyr. I think I am ideal for this job, even though I don't yet know much Zephyr nor ESP.

Shall I make notes of what I did to get a working development environment, so you both (and others) can comment and/or use my notes to set up your development environment?

[pinkit-cwa]

First I think the approaches differ a bit (correct me if I am wrong): I want to create a full device without the need of any middle-ware. I am still not sure if this works out, because of the very limited memory, but I think with a relay server putting the Exposure Key exports into much smaller chunks it should be generally be possible. I think the bracelet approach will need some middle-ware because the capabilities are even more limited than the ESP32. The choose of ESP32 came to: it is cheap, has WiFi and BLE. Without WiFi a middle-ware will always be required.

I confirm that your approach is different from my vision. But it is not incompatible.

I plan to:

• port your code to Zephyr OS
• add at least two API layers for communication to CWA:
  • one API layer for devices that can just do the sending of beacons, storing of received beacons, but not the calculations for risk analysis. CWA will obtain the received beacons and perform risk calculations
  • one API layer for devices that can do all the calculations themselves, but not fetch the TEK databases themselves or display the result. CWA will forward TEK updates and obtain the risk calculation result from the device / not get any information about the encounters (I think this is what @olafland CovidBacelet needs)
  • another API to obtain the up to 14 TEK keys from the device in case you want to report an infection via CWA

Your all-in-one-device will in the end remain fully usable. And I will profit from your "over my top" layers in multiple ways: I will more easily understand your issues with the zip files and be able to help you there. I would not epect any issues, as those zip files should be decompressed in some "streaming" way, i.e. you need not uncompress the complete file, you jsut need to be able to read it one section / TEK key after the other. Only if the memory is not enough to perform the decrompression itself, then you indeed do have an issue.

But on the other hand I also think your all-in-one device is not really a "one-size-fits-all" product. For example kids should not have to care about all of this, they should enjoy life, learn things, play. Not check their covid status. Their parents should know if there is a risk that their kids are infected, so they (and not the kids) can decide what to do. For elderly the same holds, just the other wy around: their kids should care.

With these two API layes introduced, it will be easy for other people to cherry-pick those parts of the code they need, to build own products based on cheaper and more limited hardware. I want as many people as possible to use this new option of disease control, as from what I know contact tracing alone and isolation in case of contact is enough to eradicate covid, and possibly many more diseases.

Further plans are:

• an "endless tape with sync marks" like NVM file system storage, taking into account
  • FLASH
    • can only be erased in huge blocks
    • and not necessarily be written with byte granularity
    • as well as low erase endurance of just 10k cycles
  • only part of the tape can be stored. Subsequent new data erases/overwrites older data
  • fast botstrapping / mounting of the tape shall be possible, to enable frequent power-off of the RAM
  • (possibly lossy, i.e. least recent scan might not be recoverable) recovery in case of power loss during write included
  • this will be a "logging file system"

Zephyr and some other platforms already include this as "flash circular buffer".

• improving the processing of received ENF beacons:
  • reducing the time to search through previously received beacons
  • minimizing the amount of RAM needed between scans
    Idea / proposed algorithm:
  • for each scan (every 2.5 minutes or so):
    • mount the "tape"
    • go to the end of the "tape"
    • write a "sync" mark including some current time approximation to the "tape"
    • for each received ENF beacon
      • check if the received beacon was already seen in one of the previous scans (within a suitable time frame / last 30 minutes), or not
      • for all not yet seen beacons:
        • make thenm wuickly searchable by putting themn all into a cockoo filter containing these beacons. This basically is a not fully populated list containing all received beacons, sorted by some hash calculated on the beacon. In the end just two lookups per beacon are needed to see if it is contained or not.
        • include a received field strength byte (I think its called RSSI)
        • store a table of (received ENF (16+4 bytes),RSSI (1 byte)) to the "tape"
      • for all previously seen beacons:
        • build a reference to the appropriate previously received beacon, composed of a reference to the previous scan ("5 sync marks back") and the index of the beacon ("5th beacon in the not yet seen table there"). I think this should be 4 + 12 bits, enabling 16*~2.5 / up to 40 minutes of "back references", with up to 4096 ENF beacons per scan
        • store a table of (reference (2 byte),RSSI (1 byte)) pairs to the "tape"

[pinkit-cwa]

The license: besides that I am a big fan of copyleft, the main reason for me is my own lack of knowledge in the C/embedded section, so that I can learn from contributions because of copyleft. I don't want that someone finalize a device with my code and I can just gaze at the end product/binaries. Also I think it is the best fit for DIY makers, that they can easily create own devices.

Regarding the license. I think we are flexible. We choose Apache as its is the license used by Zephyr, our underlying OS. This makes it much easier when it comes to integration with the OS. Nonetheless, we care more about overalll progress than a specific license.

I agree with both of you. I also am a big fan of copyleft. Without copyleft we would not be where we are already ("standing on the shoulders of giants"). The issue with copyleft is that it prevents some companies from using your source, as they may have to hand out some of their IP / source as well - all of the combined product / derivative work.

As a company, I am also a big fan of Apache style ("open source") licenses or public domain, where I do not have to disclose my IP, and in some case not even need to mention what my product is based on.

In case of a "contact tracing" application like "Exposure Notifications", I sure want to see the source code of the device, and also want to make sure I can resolve potential privacy issues by chaning the source or adjust the device to my personal needs. Also I sure want to meet all the CCC criteria for contact tracing. So my clear preference for this project is copyleft.

[pinkit-cwa]

2\. According to ENF specs, if possible, the BLE MACs shall be randomized, and this shall happen in sync with the changing of the sent exposure notification beacons. I did not see tat in CovidBeacon (please confirm), but it is contained in ESP-ENA. In this context, can you quickly point me to some function of Zephyr that can be use to change the BLE MAC address? If not I'll find out myself, but if you can quickly answer this question, then I save some time.

@olafland From your very nice demo video found in README.md, I see that this is already implemented. So I just overlooked that part in the source code. Great!

[pinkit-cwa]

@olafland @Lurkars

Success! I was able to build CovidBracelet.

tldr
The build command is:
$ cd ~/zephyrproject
$ west build -p auto -b nrf52840dk_nrf52840 $(path to CovidBracelet)
/tldr

Installing Zephyr as documented in the "getting started" section of www.zephyrprojct.org worked like a charm.

After I installed it, I thought, well to find out how to directly invoke cmake. That was a bad mistake and took me hours of debugging: How do I name a board? And why do I get a

fatal error: nrfx.h: No such file or directory
? Well, after that episode I continued to read at (5) Build the Blinky Sample. Based on what I read there, I tried:

$ cd $(parent of CovidBracelet directory)
$ west build -p auto -b nrf9160dk_nrf52840 CovidBracelet/

which complained that I cannot build, I first have to run one out of 8 possible other commands.

Well, I tried:
$ west init
but soon stopped that as it probably would have checked out another instance of zephyproject. Removed newly created .west subdiretory.

Well, how is it in the blinky sample? There, you remain in the zephyrproject directory. So I did a
$ cd ~/zephyrproject
$ west build -p auto -b nrf9160dk_nrf52840 ../bracelets/CovidBracelet/

which complained with:

#error "Unsupported board: led0 devicetree alias is not defined

I tried various boards but all failed. So I brute-forced the board by trying until success:

$ for board in adafruit_feather_nrf52840 nrf52832_mdk nrf52833dk_nrf52820 nrf52833dk_nrf52833 nrf52840_blip nrf52840_mdk nrf52840_papyr nrf52840dk_nrf52811 nrf52840dk_nrf52840 nrf52840dongle_nrf52840 nrf52_adafruit_feather nrf52_blenano2 nrf52_sparkfun nrf52_vbluno52 nrf52dk_nrf52810 nrf52dk_nrf52832 nrf9160dk_nrf52840 thingy52_nrf52832 nrf52_bsim ; do rm -rf build; west build -p auto -b "$board" ../bracelets/CovidBracelet/ && echo "$board" && break; done

In the end, all failed. But, for some I saw no #error, just an

error: 'sys_csrand_get' undeclared
. Well that ougth to be fixable. I searched headers for this function and found it in <random/rand32.h>. I added the include to main.c and retried my brute-forcing.

Which worked and printed the board name: nrf52840dk_nrf52840

[pinkit-cwa]

Further plans are:

* an "endless tape with sync marks" like NVM file system storage, taking into account
  
  * FLASH
    
    * can only be erased in huge blocks
    * and not necessarily be written with byte granularity
    * as well as low erase endurance of just 10k cycles
  * only part of the tape can be stored. Subsequent new data erases/overwrites older data
  * fast botstrapping / mounting of the tape shall be possible, to enable frequent power-off of the RAM
  * (possibly lossy, i.e. least recent  scan might not be recoverable) recovery in case of power loss during write included
  * this will be a "logging file system"

@olafland that is, unless Zephyr already contains somethibng like this?

Zephyr contains something like this. It is called Flash Circular Buffer, and looks suitable for storing of received beacon information.

[olafland]

Correct, this should be perfect for storing the keys. Just, the one you linked is part of MyNewt and not Zephyr and so might need porting. Nonetheless, maybe Zephyr has its own version, would need to check.

…

On 20. Aug 2020, at 09:50, pinkit-cwa ***@***.***> wrote: Further plans are: * an "endless tape with sync marks" like NVM file system storage, taking into account * FLASH * can only be erased in huge blocks * and not necessarily be written with byte granularity * as well as low erase endurance of just 10k cycles * only part of the tape can be stored. Subsequent new data erases/overwrites older data * fast botstrapping / mounting of the tape shall be possible, to enable frequent power-off of the RAM * (possibly lossy, i.e. least recent scan might not be recoverable) recovery in case of power loss during write included * this will be a "logging file system" @olafland that is, unless Zephyr already contains somethibng like this? Zephyr contains something like this. It is called Flash Circular Buffer, and looks suitable for storing of received beacon information. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.