add script to compress firmware before ota upload

[dkneisz]

Description:

adds an extra script to compress firmware bofore OTA upload. The Script is located in scripts/compressed_ota.py and added in platformio.ini with "extra_scripts = scripts/compressed_ota.py". This fixes the issue that OTA failes for Sonoff RF Bridge due to the code becoming too large with the last releases. In my case the compression reduced the firmware to 72%.

Checklist:

• The pull request is done against the latest development branch
• Only one feature/fix was added per PR and the code change compiles without warnings
• I accept the DCO.

[dkneisz]

Unfortunately this does not fully solve the OTA issue for Sonoff RF Bridge. The bin-file od the current develop branch seems to be so big, that there is no space even for the compressed bin-file. It worked one time updating from a previous version. Also it looks like this is not working for ESP32 (at least for me it started uploading and I got an error after 8% or so). Anyways I reworked the script a little bit to also compress on build.

[1technophile]

If you remove MQTTsetMQTT macro are you still unable to OTA update the RFBridge ?

[dkneisz]

Yes, even if i remove MQTTsetMQTT it's still too large. Unfortunatly I had no time to investigate any further this weekend.

[DigiH]

When investigating for the pull request #1046 I found that disabling MQTTsetMQTT only creates a small space saving, similarly to only disabling MQTT_HTTPS_FW_UPDATE. Only disabling both resulted in a viable reduction which enabled the OTA update for the Avatto again. Maybe this might also help here.

Here are the results from my tests:

0.9.8
Checking size .pio/build/avatto-bakeey-ir/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [==== ] 42.4% (used 34772 bytes from 81920 bytes)
Flash: [====== ] 55.2% (used 528716 bytes from 958448 bytes)

0.9.8 with just
'-UMQTTsetMQTT'

Checking size .pio/build/avatto-bakeey-ir/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [==== ] 42.4% (used 34732 bytes from 81920 bytes)
Flash: [====== ] 55.0% (used 527296 bytes from 958448 bytes)

0.9.8 with just
'-UMQTT_HTTPS_FW_UPDATE'

Checking size .pio/build/avatto-bakeey-ir/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [==== ] 40.6% (used 33236 bytes from 81920 bytes)
Flash: [===== ] 53.7% (used 515044 bytes from 958448 bytes)

0.9.8 with
'-UMQTTsetMQTT'
'-UMQTT_HTTPS_FW_UPDATE'

Checking size .pio/build/avatto-bakeey-ir/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [==== ] 39.9% (used 32684 bytes from 81920 bytes)
Flash: [==== ] 42.3% (used 405024 bytes from 958448 bytes)

[dkneisz]

@DigiH sorry for the late reply. I already disablet both options, and this is what I get:

Checking size .pio\build\rfbridge-PCRoom-remote\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [====      ]  38.3% (used 31348 bytes from 81920 bytes)
Flash: [======    ]  59.1% (used 450484 bytes from 761840 bytes)
Creating BIN file ".pio\build\rfbridge-PCRoom-remote\firmware.bin" using "C:\Users\derda\.platformio\packages\[email protected]\bootloaders\eboot\eboot.elf" and ".pio\build\rfbridge-PCRoom-remote\firmware.elf"
compressFirmware([".pio\build\rfbridge-PCRoom-remote\firmware.bin"], [".pio\build\rfbridge-PCRoom-remote\firmware.elf"])
Compressing firmware for upload...
Compression reduced firmware size to 72% (was 454640 bytes, now 329421 bytes)

I'm using hacked Sonoff RF bridges with "ZgatewayRF". It's only about 17kB too much for OTA falsh.

I also set 'ESPWifiManualSetup="true"'.

[DigiH]

@dkneisz - what a shame about the measly 17kB causing OTA to still fail :(

While I'm not really familiar with the Sonoff RF bridge, wouldn't increasing the partition size with the custom setting of

board_build.ldscript = eagle.flash.1m64.ld

for the ESP8285 help as well, as with the Avatto ESP8266. The default seems to be eagle.flash.1m256.ld for the boards. So the extra 192 kB should help in your case.

Unfortunately the initial upload with this has to be done via direct serial cable, but then with the extra space gain repeated OTA should be possible with the previously mentioned disabling flags and your compression script.

All the best.

[DigiH]

P.S. or possibly using and older OMG version, which used to be quite a bit smaller, with

board_build.ldscript = eagle.flash.1m64.ld

in OTA mode to increase the partition size.

Not sure if this would be feasible though ;)

[dkneisz]

@DigiH thanks for the tip, this seems to work. I was able to OTA flash one of my SRFB, that still had an older OMG version, with this setting. Flashing one, that had a newer OMG version, took some persuasion. I had to flash it multiple times and erase the flash for it to work, but in the end it is working.

[DigiH]

@dkneisz - glad to hear that OTA is working for you again now :)

@1technophile - maybe board_build.ldscript = eagle.flash.1m64.ld should also be included as a default for all Sonoff bridge environments, as for the Avatto, or any other ESP8266 1MB based boards which are not easily connectable. The saved 192 kB really seems to make a difference for OTA updates and I haven't seen any negative effects with the changed partitions on any of my Avattos.

[dkneisz]

I implemented some changes to my solution:

• the script is now called "compressFirmware.py" instead of "compressed_ota.py" because it now compresses the firmware in when it it build.
• I commented out the line in platformio.ini under [platformio] because this seems not to be working in any case:

;extra_scripts = scripts/compressedFirmware.py ;uncomment this to compress the firmware. This helps updating e.g. Sonoff RF Bridge via OTA flash by saving space for the uploaded firmware.

• I added examples for hacked rfbridge to "prod_env.ini.example". This included one for serial falshing, using board_build.ldscript = eagle.flash.1m64.ld and one for ota flashing using extra_scripts = scripts/compressedFirmware.py

[1technophile]

@1technophile - maybe board_build.ldscript = eagle.flash.1m64.ld should also be included as a default for all Sonoff bridge environments, as for the Avatto, or any other ESP8266 1MB based boards which are not easily connectable.

I agree, @dkneisz maybe you could update directly platformio.ini sonoff rf bridge envs, add this ldscript
and refer into the env with a comment to your script (example: you need to uncomment the following line if you are having troubles with OTA)

[dkneisz]

@1technophile I added board_build.ldscript and extra_scripts to both of the rfbridge environments in platformio.ini. The extra script is commented out. I didn't choose this option at first, because the script mainly helps for OTA flashing, and there is no environment defined for this in platformio.ini.

It also made no sense to me to define such an environment there, because you need to add the ip address of your device. So you would have to use a placeholder, that would make platformio.ini invalid and it would not pass your checks (on github), or use a dummy ip. That's why I added the examples to prod_env.ini.examples.

[1technophile]

Thanks, I get it

[mcspr]

Just to note, in case of update to 3.0.0 there's a built-in solution right in the framework builder:
esp8266/Arduino#7727
.ini needs to have build.gzip_fw property set to true