v5.1.0 broked something related to LittleFS and SPIFFS with TinyPICO board... works in Arduino IDE.

[sblantipodi]

This is a small code sample that read and write to Little FS on ESP32.

#include "WiFi.h"
#include <FS.h>
#include <LittleFS.h>

void setup() {
  Serial.begin(115200);
  // Set WiFi to station mode and disconnect from an AP if it was previously connected
  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  delay(100);
  Serial.println("Setup done");
  if (!LittleFS.begin(true)) {
    Serial.println("Formatted");
  } else {
    Serial.println("Formatteddd");
    LittleFS.format();
  }
  File file = LittleFS.open("/prova.txt", "w");
  //Write to the file
  file.print("dsadasdsa");
  delay(1);
  //Close the file
  file.close();
  Serial.println("wrote");
  File file2 = LittleFS.open("/prova.txt", "r");
  if (!file2) {
    Serial.println("No Saved Data!");
  }
  Serial.println(file2.read());
}

void loop() {
  Serial.println("scan start");

  // WiFi.scanNetworks will return the number of networks found
  int n = WiFi.scanNetworks();
  Serial.println("scan done");
  if (n == 0) {
    Serial.println("no networks found");
  } else {
    Serial.print(n);
    Serial.println(" networks found");
    for (int i = 0; i < n; ++i) {
      // Print SSID and RSSI for each network found
      Serial.print(i + 1);
      Serial.print(": ");
      Serial.print(WiFi.SSID(i));
      Serial.print(" (");
      Serial.print(WiFi.RSSI(i));
      Serial.print(")");
      Serial.println((WiFi.encryptionType(i) == WIFI_AUTH_OPEN) ? " " : "*");
      delay(10);
    }
  }
  Serial.println("");

  // Wait a bit before scanning again
  delay(5000);
}

if I use a TinyPICO board, this small snippet fails reading writing LittleFS when using
[email protected]
it works well when using
[email protected]

I noticed that the only way to make it work with
[email protected]
is to use
board_build.flash_mode = qio
instead of
board_build.flash_mode = dio

but this does not have sense to me.

when using [email protected]
that snippet works with both QIO and DIO.

the problem happen on TinyPICO board only, it does not happen on LolinD32 or other ESP32 boards I have.

Same snipped always works well with Arduino IDE when using Core 2.0.4.

[Jason2866]

Same as #867
@valeros I still think it is a good idea to change the default flash mode to qio to all newer ESP32x MCUs.

@sblantipodi There was a change/fix in the way how the Arduino Bootloaders are compiled in IDF for Arduino for core 2.0.4.

[sblantipodi]

@sblantipodi There was a change/fix in the way how the Arduino Bootloaders are compiled in IDF for Arduino for core 2.0.4.

thanks for the answer and for joining in @Jason2866
you are referring to a "change/fix", is there a way I can fix this problem?

[Jason2866]

@sblantipodi First try flash mode qio for all boards you have. This should fix when flashing via Platformio. For generating a valid factory firmware you have to do changes.
Discussed here

[sblantipodi]

@sblantipodi First try flash mode qio for all boards you have. This should fix when flashing via Platformio. For generating a valid factory firmware you have to do changes. Discussed here

I noticed it, QIO is the flash mode needed to work well on all boards now.
Thanks!!!

[Jason2866]

@valeros Looking at this https://github.com/espressif/arduino-esp32/blob/master/tools/platformio-build-esp32.py#L306-L313 there is always the qio lib included since the variable there is not set from anyware. Or do i miss here something?

[TD-er]

So then we could also set build.arduino.memory_type to dio_qspi in the board definitions?

See for other other options: https://github.com/espressif/arduino-esp32/tree/master/tools/sdk/esp32

Something like this:

{
  "build": {
    "arduino":{
      "ldscript": "esp32_out.ld",
      "memory_type": "dio_qspi"
    },
    "core": "esp32",
    "extra_flags": "-DARDUINO_ESP32_DEV -DARDUINO_USB_CDC_ON_BOOT=0 -DESP32_16M",
    "f_cpu": "240000000L",
    "f_flash": "40000000L",
    "flash_mode": "dio",
    "mcu": "esp32",
    "variant": "esp32",
    "partitions": "esp32_partition_app4096k_spiffs8124k.csv"
  },
...

[Jason2866]

yep. But for all except the ESP32-S3 it could be generated from flash_mode since it is always the same. Only the ESP32-S3 can have different variants.
Imho it should be done that way, so the existing boards manifest are still useable/valid.
For the S3 the entry is needed most of the time since there are some variants possible
(Probably this is the reason why many users dont get the S3 going with PSRAM).

[TD-er]

OK, this really has become my new nightmare....

I tested myself to make a build where I set the memory_type to dio_qspi along with the flash_mode to dio as shown above.

• OTA update on board previously flashed with flash_mode set to DOUT => Boots fine, but in DOUT mode, while build was set to use DIO
• OTA update on board previously flashed with flash_mode set to DIO => Boots fine, in DIO mode as expected.
• Serial update on same board I just OTA updated in previous step => Boots in DIO mode, but LittleFS filesystem is gone!!!

N.B. the old builds apparently must have had qio_qspi set as memory_type as these boards all have been flashed in the past few weeks during development.

[cyberman54]

This issue also seems to affect Arduino Preferences class?
It does not work any more after ugrading v5.0.0 to v5.1.0

[valeros]

@Jason2866

Looking at this https://github.com/espressif/arduino-esp32/blob/master/tools/platformio-build-esp32.py#L306-L313 there is always the qio lib included since the variable there is not set from anyware. Or do i miss here something?

TBH, I'm a bit unaware why the Espressif team decided to use the qio_qspi by default. What is correct behavior here? Should the build script dynamically configure the memory_type field according to flash_mode?

[TD-er]

@valeros See also my findings here: letscontrolit/ESPEasy#4200

TL;DR
Changing the qio_qspi to dio_qspi does still seem to work fine when I OTA update a board with LittleFS.
But flashing the same board via serial and thus updating the bootloader too, resulted in an unmountable file system.

[Jason2866]

@valeros I did tests this morning regarding to. When i changed the setup to memory_type the same as flash_type i had no boot loops nor a defect LittleFS.
When the are not align it did not always work.
BUT until now i never managed to get the device in flash mode qio.
Tbh i have no idea why the second stage bootloader always enables only dio
or when used qout it always ends in running mode dout.
Btw. opened #880 small changes/fixes for flash modes. I think not all changes are there in your latest commit

[TD-er]

Tbh i have no idea why the second stage bootloader always enables only dio

Not sure if that's true.
I have no idea how to check the actual flash mode being used. It looks like the function to get the mode is reading a header. But I am not sure if that's a guarantee to be the same as the runtime mode used on the flash chip.

[Jason2866]

@TD-er True! Good point.
EDIT: TADA You where right!!! Great team work. You pointed me in the right direction.
Just compiled a new framework and enabled IDF bootlogging:

Build:Feb  7 2021
rst:0x1 (POWERON),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5810,len:0x1278
load:0x403cc710,len:0xb9c
load:0x403ce710,len:0x2b50
SHA-256 comparison failed:
Calculated: f526bbbf3f0d384062c796362385b439c9aa329972519fc8905554dff52750c9
Expected: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Attempting to boot anyway...
entry 0x403cc710
I (48) boot: ESP-IDF v4.4.3-145-g9ce849ce72 2nd stage bootloader
I (49) boot: compile time 11:08:46
I (49) boot: chip revision: 3
I (49) boot_comm: chip revision: 3, min. bootloader chip revision: 0
I (55) qio_mode: Enabling default flash chip QIO
I (59) boot.esp32c3: SPI Speed      : 80MHz
I (63) boot.esp32c3: SPI Mode       : QIO
I (67) boot.esp32c3: SPI Flash Size : 4MB
I (71) boot: Enabling RNG early entropy source...
I (75) boot: Partition Table:
I (78) boot: ## Label            Usage          Type ST Offset   Length
I (84) boot:  0 nvs              WiFi data        01 02 00009000 00005000
I (90) boot:  1 otadata          OTA data         01 00 0000e000 00002000
I (97) boot:  2 safeboot         factory app      00 00 00010000 000d0000
I (103) boot:  3 app0             OTA app          00 10 000e0000 002d0000
I (110) boot:  4 spiffs           Unknown data     01 82 003b0000 00050000
I (117) boot: End of partition table
I (120) esp_image: segment 0: paddr=000e0020 vaddr=3c100020 size=375c0h (226752) map
I (127) esp_image: segment 1: paddr=001175e8 vaddr=3fc8cc00 size=02dfch ( 11772) load
I (135) esp_image: segment 2: paddr=0011a3ec vaddr=40380000 size=05c2ch ( 23596) load
I (144) esp_image: segment 3: paddr=00120020 vaddr=42000020 size=f41d4h (999892) map
I (150) esp_image: segment 4: paddr=002141fc vaddr=40385c2c size=06f40h ( 28480) load
I (159) esp_image: segment 5: paddr=0021b144 vaddr=50000010 size=00010h (    16) load
I (165) boot: Loaded app from partition at offset 0xe0000
I (170) boot: Disabling RNG early entropy source...

00:00:00.000-253/08 HDW: ESP32-C3 
00:00:00.010-252/07 UFS: FlashFS mounted with 288 kB free
00:00:00.024 CFG: Loaded from File, Count 32
00:00:00.068 BRY: Berry initialized, RAM used=4215 bytes
00:00:00.075 Project sonoff - Tasmota Version 12.0.2.4(UNIVERSAL)-2_0_5_dev(2022-08-16T13:35:38)
00:00:00.097 TAP: Loaded Tasmota App 'Partition_Wizard.tapp'
00:00:00.959 WIF: Connecting to AP1 Jason_Home_WLAN Channel 11 BSSId 00:A0:57:2A:BD:19 in mode 11n as sonoff-A44FC0-4032...
00:00:06.064 WIF: Connected
00:00:06.317 HTP: Web server active on sonoff-A44FC0-4032 with IP address 192.168.2.178

Why i didnt do that earlier???

[Jason2866]

Opened issue espressif/arduino-esp32#7140

[TD-er]

Ah great :)
Yep good teamwork 👍

I'm curious whether my unit here may indeed not support QIO but only QOUT.
And also on what basis the 2nd stage bootloader decides to switch on QIO or perhaps QOUT.

[Jason2866]

This issue also seems to affect Arduino Preferences class? It does not work any more after ugrading v5.0.0 to v5.1.0

@cyberman54 yes, since including the wrong SPI lib driver results in damaging flash content. This is hitting LittleFS, SPIFFS, NVS and all code stuff writing to flash.

[valeros]

I've updated the Arduino package for ESP32 with the patch by @Jason2866 submitted in espressif/esp32-arduino-lib-builder#80

[Jason2866]

@valeros Great, but why you removed the env.BoardConfig().get(\"build.arduino.memory_type\" part from my PR in your build? For the S3 it is still needed to adjust the possible Flash/PSRAM variants.
My idea was mainly to fix ESP32, C3 and S2 without the need to change the boards manifests for this MCUs.
To be clear the entry memory_type is needed for the S3 without there will always be issues, because the correct type of Flash/PSRAM can not be set!
(imho the introduction of opi, the way espressif did, for the S3 is a mess...)

[valeros]

@Jason2866 I just glanced at espressif/esp32-arduino-lib-builder#80 and didn't see any branches special for S3. Anyway, fixed in the latest package.

[cyberman54]

How can i use the current version with the fix with platformio?

[valeros]

@cyberman54 Just run pio platform update espressif32 in the IDE terminal.

[Jason2866]

@valeros Perfect! Thx, for your great support :-)

[Jason2866]

Are you going to open a PR for your way? It is better than mine. So i could close my PR

[valeros]

@Jason2866 Do you mean a PR to https://github.com/espressif/esp32-arduino-lib-builder ? I wasn't planning on submitting a new PR, let's see what the maintainers say about yours.

[Jason2866]

Fine, the PR does work. But it is not easy to understand at first sight was it does ;-)

[Jason2866]

@sblantipodi Please try other flash modes than qio. The work now too.
Please close the issue ;-)

[sblantipodi]

@Jason2866 Hi, I'm from mobile now, I'll try it next week as soon as I will be back home. Thanks for quoting.

[sblantipodi]

I confirm it works awesome, congrats guys!
Thank you!