Add support for i18next localisation library

[laszloh]

Playing around with i18next as localization agent. Idea extracted into own branch from webpage optimisation.
This pull request is currently untested.

ToDo List:

• Create nodejs (probably Express) test framework for local development
• Add automatic language detection
• Add all strings to English translation
• Add German translation
• Embed language and Javascript files into PROGMEM
• Test it :)

[biologist79]

Thanks for your contribution (which is really appreciated) but I've two things to mention:
a) There's already been an attempt to completely redesign the webgui (https://github.com/sonovice/espuino-ui, https://sonovice.github.io/espuino-ui/ => enter random credentials, https://forum.espuino.de/t/refactoring-des-web-interfaces-und-der-rest-api/1284). It's currently (due to lack of time) paused (whatever that means). So the question is if it makes sense to do it (somehow) twice.

b) ESPuino uses a custom flash partition-layout (https://github.com/biologist79/ESPuino/blob/master/custom_16mb_ota.csv). I'm not familiar with LittleFS so I'm not sure what it takes partition-wise. That being said: it's no option if changes to this layout are necessary as this has a really big impact (ESPuino is already used by hundrets of users).

[laszloh]

Hi,
regarding point a) yes, that definitely looks different :). And no, it does not make much sense to do it twice (expect if you are currently in dire need of i18n). My time investment is not a big deal, I also used this to get a rough overview of the project.
I'll look into the new webgui on the weekend. Unfortunately I'm not familiar with Svelte (I'm more a React guy) but I'm open to learn new things.

Add b) in the 16mb parition, you have the partition named "spiffs" (with a size of 0x330000 bytes ~3MB). Thats the internal file system of the esp. It can be formatted and used either with spiffs (not recommended) or littleFS and files written there can be accessed just like the files on the sd-card. In my projects I mostly use them for saving configuration (not a usecase here, since you use nvs).
I understand that compability is a concern, the 4MB partition image does not contain an spiffs. So putting stuff there will break that partition image. I'll keep that in mind and not use the emedded file system.

For static web pages, one can either put them into the image itself (like you are currently doing with PROGMEM). One point would be to compress the files before wirting them to the flash to save space and bandwith (so basically you are wirting the gzipped files into the esp, but then you can not templates since for the webserver they seem like a binary blobb). My first try with python was semi successfull, but with nodeJS it will be way easier.

I've already registered at the Forum (my native tongue is german), it just takes time to get used to it and navigate in there, there is tons a of information (not a bad thing ;) ).

[tueddy]

@laszloh Just a note:
Current webpage uses the ESPAsyncWebervers template processor. To compress to a static HTML file you have to get rid of template processor placeholders first..

[laszloh]

I've added the files to the PROGMEM. Compiling and comparing the master branch against my branch.
I see an increase in the flash usage by ~22kB.

The results:

master:

RAM:   [==        ]  21.9% (used 71624 bytes from 327680 bytes)
Flash: [=====     ]  53.3% (used 2061944 bytes from 3866624 bytes)

i18n:

RAM:   [==        ]  21.9% (used 71624 bytes from 327680 bytes)
Flash: [=====     ]  53.9% (used 2083800 bytes from 3866624 bytes)

The increase comes from the embedding of the 3 js libraries needed for i18next and it's supprting libraries.

The main question is, if the increase is ok. If not, I could link to cdn (like with the other js-libraries). Then we'd only see a small increase due to the json files needed for the translations (about 5k). On the downside the access point would not get any translation (or someone would have to write a really basic i18n engine for it).

The code itself is still untested, unfortunately I did not find time to do it yet.

[laszloh]

Rebase on the current master finished. I've tested it on my setup, it work in both accesspoint and normal operation mode.

[biologist79]

Tried it but no text is shown.
And Noscript indicates there's still JS-stuff requested from Cloudflare, espuino.de and code.jquery.com.

[tueddy]

First thank's for your conribution!
I've tested and have missing text too. It seems there is a missing resource "/js/loc-i18next.min.js" ?

[laszloh]

Hi,
Thank you for the reply. It is a naming error and I missed it while rebasing. Interesstingly I had to clean and rebuild to reproduce the error (I had compiler cache enabled and it messed with the file generation).

biologist79, I had only embedded the javascript libraries needed for the translation engine (since they are not avaliable at the first setup, when the ESP32 is in AP mode). Everything else is still fetched externally. If I embedd them, it will cost about 400kB flash space to embedd them.

I would rather try to make the espuino-ui work, on the long run that seems like a better idear, since it optimizes the build (by striping unused javascript functions and embedding everything into a single compressed file).

[tueddy]

I can confirm i18n is working now (just tested the management.html)!

Only some minor glitches:

• Loading/reloading the page flickers because of missing texts. Is there a way to prevent flickering?
• Buttons are a bit narrow/overlapping:
  
• Labels margin are bit of (too high). Don't know if this was before:
  

Anyway, good work and thank's for contributing,
Dirk

[biologist79]

Sorry, thought all libs would be part of the new FW. However, I'm with how it's done currently.
Will test ASAP.

Thanks for your contribution!

[biologist79]

Can confirm space between buttons could be "way" more.
Don't have any flickering issues.

However: like it 👍

[tueddy]

I noticed a small issue, the contextmenu remains always english:

[laszloh]

I totally missed, that there was a context menu :D. I fixed the translation and the layout errors (by putting in &nbsp between the buttons).

The label margins seems to have the same height in both my branch and the main branch:

(left is the i18n and right the master)

I could reproduce the flickering (sort of) by reducing the "network speed" and disabling the cache in the Developer Tools. This reduces the speed with which the translation files can be fetched and significantly increased the time it takes i18next to fill in the translation. With this I get a short flicker when the layout is updated after the files are fetched.

I'm currently thinking about putting the normal text back in and see, if i18next overwrites it. This would ableviate the flickering (since there is already text in the layout), but it also seems a bit hacky. I'll think it over today and look at it tomorrow :)

[laszloh]

Flickering should be reduced/fixed. I did not find any better soultion than putting some of the texts back in to prevent the design shifting due to delayed i18n loading.

[biologist79]

Found some minor issues:

a) Space is missing between "Steuerung" and its icon.
b) Same goes for "WLAN". However, other items are ok.
c) For some actions (e.g. RFID-card applied) there's a popup shown with text. Even with having lang=DE active, this remains English.
d) "Modifikation" should be "Modification" for lang=EN.

Thanks for your work - really appreciate it!

[laszloh]

Done, I also found some other text's in the file upload part which I translated.

[biologist79]

a) ok
b) ok
d) ok

c) Now messages appear like "toast.rfidDetect" or "toast.success" :-)

[laszloh]

I've disabled the cache since it seems that firefox is not sending the required headers (I don't know why).
Since the "If-None-Match" header is missing, it does not seem to pick up on changes.

Please try reloading with CTRL+F5 after compiling with the last commit to clear the browser cache.

[biologist79]

Merged, finally :-)
Thanks a lot for your work!

[Joe91]

Hi, can you tell me how to convert the files to get them to the "dist" folder if I modified some files in the "html" folder?

[biologist79]

Would have expected (but didn't test) this works automatically when doing a build.
However, as it turns out this doesn't work => bug.

@laszloh Please fix.
Additionaly it would be nice to have some inline documentation placed inside of processHtml.py.

[laszloh]

@Joe91 @biologist79 I knew I had forgotten about something. I did the compression of the files manually (so creating the minifed version & the zips), since I had not found a good python html minifier.

Finding a minifier, which I could get to work was a real pia, but at last I found one, which seems to work with the setup.
I created a bugfix branch, which now uses the content of the "html" folder directly to create the needed html file headers.

Should I create a pull request, or can you directly merge fef0288?

[biologist79]

No, I don't need a PR. Can fetch this directly by cherry-picking.
Thanks for your support. Being a "PERLer" I'm somehow naked when it comes to python 🤣