travis-ci.org is shutting down

[mcspr]

Quoting most recent CI job https://travis-ci.org/github/letscontrolit/ESPEasy/builds/753527504

Please be aware travis-ci.org will be shutting down in several weeks, with all accounts migrating to travis-ci.com. Please stay tuned here for more information.

I have not found any issues even mentioning it, and idk if this was discussed internally or on forums, but simple "travis-ci.org" search did not find anything. Also, I assume this is the correct place for the issue / discussion, and not the forum.
Some more info:

• https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
• https://blog.travis-ci.com/oss-announcement
• https://docs.travis-ci.com/user/migrate/open-source-repository-migration

Q. What will happen to travis-ci.org after December 31st, 2020? #
A. Travis-ci.org will be switched to a read-only platform, allowing you to see your jobs build history from all repositories previously connected to travis-ci.org.

Although, it is apparent that is not the case just yet, but very likely it will be in the near future

Very quick solution is to just use Github Actions, however build scripts will need to be changed (e.g. depending on any TRAVIS_... env variables or doing anything with it's API). Which I could try in the PR, unless some other solution is in the works

[TD-er]

Yep this for sure is going to be a problem as Travis.com is a paid service with a free plan which for sure is not going to cover our needs.
The free plan will allow for about 7 - 8 builds a month, which we sometimes use on a single day.

I was looking into hosting myself a Jenkins build server for this (already have a build server setup for "custom builds" which I'm working on)
But if transitioning to Github actions is also an option, then be my guest...
One of the truly frustrating things about Travis.org lately was that it may take upto 48h for a build to complete.
The last few weeks it's performing a lot better, but like a month ago it was really unusable.
Not sure how quickly Github actions will complete a job?
I can imagine build times may rise if it is going to be the default CI/CD platform for many projects and I don't know when they will start demanding a paid subscription.

On my build server I can build an entire set in roughly 15 - 20 minutes, maybe even faster if I split the job over several build agents. (now the PIO envs are build in a single sequence, so half of the time is spent in linking)

N.B. the build server has been paid for by the Patreon money Jimmy and I receive.

[mcspr]

GH build time is about as fast as Travis, queue and network speeds are superb though.

For example:

• https://github.com/mcspr/ESPEasy/actions
• https://github.com/mcspr/ESPEasy/blob/ghactions/.github/workflows/build.yml
• https://github.com/mcspr/ESPEasy/runs/1694192972

As it turns out, none of the used scripts use TRAVIS_..., what I noticed via git-grep was not even used by the CI flow. However:

• It is unclear whether docs packages should be installed when not on 'deploy' / tagged build
• Caching is implemented, but depends on $somefile contents. It is unclear how to depend on vendored libs as well as the pio's lib_deps or requirements.txt, for example. There are also multiple platformio .ini files, do I need to use _base.ini ?
• PIO script seems to extract branch name instead of commit, and github checks out the tree a bit differently than travis:

GIT_HEAD: 'ghactions_7323437'

• PUYA patcher script expects ~/.platformio/packages/framework-arduinoespressif8266*/ glob, which is not the case when there is only a single package. I think it can be safely removed, as it is already upstream?

Sidenote:
Actions actually has concept of an artifact, so it is possible to attach the resulting .bin / .elf / whatever to the build job itself:
https://docs.github.com/en/free-pro-team@latest/actions/guides/storing-workflow-data-as-artifacts

Sidenote2:

RE: build server, it is possible to use a custom 'runner' for the Actions, and at least one on a free plan:
https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners
https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners

But, I'm out of my depth here. And one concern is:

Forks of your public repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow.

Which is also true for any other self-hosted solution.

[TD-er]

The platform.ini file includes the others.

Building docs is not needed for a normal test build as we also don't do anything with its build output return value.

Caching is a bit tricky now we concatenate files in some directories to overcome PlatformIO issues on Windows.
Maybe that action should only be done on Windows platforms as it also adds to the build time since the concatenated file is always 'touched' and thus all files in it will be compiled again.
Also caching should be done per env and not for all using the same cache.
It may also be a bit tricky for packages, unless we explicitly call pio update in the build script.

PUYA patches can be removed as we also do not build older core library versions that do not include the PUYA patches.

I will look into the concept of 'runner' for the GH Actions.

I will also look into the security concern you mention as it is indeed something potentially serious.
We now also run Python scripts in the repo itself, which is tricky indeed.

[TD-er]

I noticed you're uring a fork based on a very old commit...

This branch is 2432 commits behind letscontrolit:mega.

So maybe you can also test it on a more recent commit?

[mcspr]

I noticed you're uring a fork based on a very old commit...

This branch is 2432 commits behind letscontrolit:mega.
So maybe you can also test it on a more recent commit?

Different tree! Github default branch is indeed the very old mega, see the ghactions instead.

Caching is a bit tricky now we concatenate files in some directories to overcome PlatformIO issues on Windows.
Maybe that action should only be done on Windows platforms as it also adds to the build time since the concatenated file is always 'touched' and thus all files in it will be compiled again.
Also caching should be done per env and not for all using the same cache.
It may also be a bit tricky for packages, unless we explicitly call pio update in the build script.

CI only has Ubuntu build though? There are some syntax tricks that can check for the $ENV contents and select a different cache which I will try. I guess hashFiles('platformio*.ini)` would do.

Note about the pio update - platformio/platformio-core#3770 (latest comments)
I may uninstall some stuff unintentionally. Technically, PIO should manage dependencies for build automatically anyway, and for the git specifically one should probably just do a cd $GIT_DIR && git pull instead of the magic it tries to do.

Will put the docs & puya aside though (at least for now), thanks for the clarification.

[TD-er]

Thank you for testing/trying as it removes a bit of the time critical burden of moving the CI/CD due to closing of Travis.org

[mcspr]

No problem 👍

https://github.com/mcspr/ESPEasy/runs/1695629152?check_suite_focus=true
https://github.com/mcspr/ESPEasy/runs/1695629183?check_suite_focus=true
Note the caching picks up environment name and selects the ~/.platformio cache accordingly.

Another thing, I missed the = -D BUILD_GIT='"${sysenv.TRAVIS_TAG}"' (...was searching for TRAVIS_COMMIT) in the platformio.ini build_flags=..., but there does not seem to be an appropriate replacement looking at the list here https://docs.github.com/en/free-pro-team@latest/actions/reference/environment-variables

[TD-er]

Hmm that would be nice if we can get the tag from the build environment.
Right now I'm doing Python stuff to get the branch from Git and replace it in PlatformIO
See: https://github.com/letscontrolit/ESPEasy/blob/mega/tools/pio/generate-compiletime-defines.py

[mcspr]

With tags what it looks like:

• to be called on a tag, github actions needs on matching create
• when github event is create i.e. tag push for the commit, job will set GITHUB_REF environment var to the refs/tags/blah (also could match for the specific 'step' via if: ${{ github.event_name == "create" }})
• some kind of script can match ^refs/tags/ and simply use GITHUB_REF[10:]

e.g.

def get_tag():
    # since " If neither a branch or tag is available for the event type, the variable will not exist."
    ref = os.environ.get("GITHUB_REF", "")
    if ref and ref.startswith("refs/tags/"):
        return ref[len("refs/tags/"):]

    return ""

env.Append(CPPDEFINES=[("BUILD_GIT", "\\\"{}\\\"".format(get_tag()))]

https://github.com/mcspr/ESPEasy/runs/1697632179?check_suite_focus=true#step:7:153
(job created via a dummy git tag -a test-1021171 -m "test" + push. it still builds the normal commit at the same time though, which seems like a waste)

[TD-er]

Please have a look at the Python script I made for setting compile time defines in the C++ code.
Now that I see your \\\" code, it reminds me of the struggle I had to make it work in both Windows and Linux.
Also I stopped using env.Append(CPPDEFINES=... in PlatformIO as it doesn't work very predictable.
I now use env.Append(BUILD_FLAGS=custom_defines) (see tools/pio/pre...py) which does result in predictable and reproducible compile behavior on all code.
Using the CPPDEFINES route is causing some defines not to be defined in all code that's being compiled.

[mcspr]

Do you mean env vs. projenv in normal scripts, or something else? afaik, since I append in pre:script, it gets cloned into both
Quoting is really weird, true :) I wonder why scons does not manage this automatically, somehow

[TD-er]

I think it has something to do with handling command line arguments, so it is OS dependent.

Appending defines in the projenv scripts indeed.
I used CPPDEFINES before and ran into issues where defines appeared not to be set in all code during compilation.
When appending the defines to the already existing BUILD_FLAGS it does work fine. Note you need to prefix the defines with -D
Not sure if it is a bug, or something I do not (yet) understand.

[mcspr]

So the BUILD_FLAGS is basically either build_flags = ... for the env, or the PLATFORMIO_BUILD_FLAGS=... os environ variable, meaning we do need command line switches present. Original value is set via .ini, os environ gets appended right after it:
https://github.com/platformio/platformio-core/blob/9c20ab81cb68f1ffb7a8cac22ce95c4c797643ec/platformio/project/options.py#L337-L347
https://github.com/platformio/platformio-core/blob/9c20ab81cb68f1ffb7a8cac22ce95c4c797643ec/platformio/project/config.py#L219-L220
https://github.com/platformio/platformio-core/blob/9c20ab81cb68f1ffb7a8cac22ce95c4c797643ec/platformio/builder/tools/platformio.py#L124

CPPDEFINES though is something that gets generated some time into the build from the BUILD_FLAGS, which then gets injected into the env object variables. e.g.

>>> env.ParseFlags("-DBLAH")["CPPDEFINES"]
["BLAH"]

fwiw I have used this approach to inject commit sha which seems to work so far across all the platforms via projenv.Append (since we don't really need this in the global flags)

edit: Still TODO - modify the github action to upload the resulting files. Will try to do that some time today (without getting distracted with pio internals research...)

[mcspr]

I also noticed the jobs list is hard-coded into CI and my previous approach to detect cache via env name does not always work.
https://github.com/mcspr/ESPEasy/actions/runs/487782572 is an example where goes through the platformio.ini files (every one of them), picks env name and platform and automatically creates a list of 'jobs' to build

There's also could be any kind of enable / disable list in place of the code, when envs do not need to be built.

I will go on generating artifacts per-job, since I can't exactly generate .zip from each env at once, like the existing build work via the build_ESPeasy.sh, and munge them together at the end to upload a release asset.
One problem though is that the official action that uploads assets is DOA - https://github.com/actions/upload-release-asset/issues
Not a big problem, just a bit annoying.

General idea is to have:

• ESP8266_ELF.zip
• ESP32_ELF.zip
• ESP8266_BIN.zip - with uploader tools
• ESP32_BIN.zip - with uploader tools
• DOCS.zip

idk if it is really worth generating source package when being hosted on github? it allows to download the 'tag' tree as an archive already, i.e.:
https://github.com/letscontrolit/ESPEasy/archive/mega-20210114.zip
(plus, we still miss what gets installed in the .pio/libdeps/$env anyway, and there does not seem to be any more build-time generated content)

[TD-er]

This is the best moment to rethink what is the best approach for the build artifacts we like to create.
Maybe it is also a good idea to generate .bin files as separate artifacts and if really desired we can also add a final step to fetch all generated bins and create a ZIP.
So that step is then depending on the build steps.

Also ReadTheDocs does support tags (and thus versions) of the docs, so that's also something to look at to generate new tags/versions in RTD when a new tag is set.
(doesn't have to be done right now)

Jimmy (@Grovkillen) is hard working on improving the flasher so it will support also ESP32 and can also be adapted to download the separate bin files from GitHub if we have them as separate files.

Maybe we can also create a link to a tagged version of the source, so you don't actually host the source ZIPs on GitHub, but rather let it generate when needed.

About the cache functionality.
I guess caching the pip installed libraries and the PIO frameworks/libdeps is the most valuable to cache.
Build dirs should be cached only if we are absolutely sure it will not lead to unpredictable builds, which was a (major) problem in the past.

[mcspr]

This is the best moment to rethink what is the best approach for the build artifacts we like to create.
Maybe it is also a good idea to generate .bin files as separate artifacts and if really desired we can also add a final step to fetch all generated bins and create a ZIP.
So that step is then depending on the build steps.

Having per-prefix builds - test_..., custom_..., hard_... and etc., also came to mind, we don't necessarily need that many parallel jobs. And it may still generate artifacts from build results that succeeded.

Job artifacts are strictly temporary though, release asset is the best approach imo. But, for example, it allows PRs to have some runnable builds.

About the cache functionality.
I guess caching the pip installed libraries and the PIO frameworks/libdeps is the most valuable to cache.
Build dirs should be cached only if we are absolutely sure it will not lead to unpredictable builds, which was a (major) problem in the past.

Cache right now only keeps the platform tools, toolchain and the framework files. Will add pip. Libs are another story, since PIO keeps them per-env. But, it's possible to install libs in pre:script to a shared location or globally to avoid library manager fetching them in the main step:
https://docs.platformio.org/en/latest/projectconf/section_env_library.html#lib-extra-dirs
https://github.com/xoseperez/espurna/blob/3a34d555d58329eebeca4ad90ce1846fa55211f1/code/scripts/pio_pre.py#L40-L55
https://github.com/xoseperez/espurna/blob/3a34d555d58329eebeca4ad90ce1846fa55211f1/code/scripts/pio_pre.py#L140

But, I am not familiar with flasher tool usage of github assets & where the source url may be used / referred to. The approach right now is to build call ./release script and push .zip files manually from the jenkins?

[mcspr]

So, at least upload & download of artifacts can be detected in bulk:
https://github.com/mcspr/ESPEasy/actions/runs/499434317
https://github.com/mcspr/ESPEasy/releases/tag/test-123469

The workflow right now is:

• check if event is 'push' starting with refs/tags/, run docs & extra tools packers. these will be job artifacts
• generate env matrix for the further builds
• build envs, create build artifacts with .bin + .elf
• last step only called if event is 'push' starting with refs/tags/ as well. it fetches all artifacts, creates a release from the tag name and message and uploads every .zip archive

Some side notes:

• surprisingly, GITHUB_TOKEN is automatically generated for the repo, no need to use hard-coded secret like travis yml did
  https://docs.github.com/en/actions/reference/authentication-in-a-workflow#about-the-github_token-secret
• I got stuck with release message. git (see workflow step in release-files) returns commit message instead of the annotated tag's one I create 🤷 I wonder what I am missing here
• BUILD_GIT & SET_BUILD_GIT_HEAD are doing almost the exact same thing, but pygit2 allows to retrieve branch + tag without git CLI tools installed (i.e. on the user machine. still needs pip install though)
• pygit2 wheel for python 3.9 is missing and it needs libgit2 devel headers

[TD-er]

Looking good.
I have to take a look at it when my head is a bit more clear, but so far I think I can follow your steps and also the Actions UI looks quite intuitive.

[mcspr]

re. pygit2, the issue comes from:

ESPEasy/requirements.txt

Line 25 in 4747dd8

pygit2==1.2.1

https://pypi.org/project/pygit2/#files
the current 1.4.0 does have the 3.9 pre-built wheels

I can also switch to a PR after all and add more envs to the build list

[TD-er]

Yes please do.
The requirements.txt is mainly based on "this works, so freeze it" using pip freeze > requirements.txt.

Every now and then I just upgrade the packages to the then current version and test if it still works.

So I guess now is the time for a new upgrade of Python packages.

A PR will be much appreciated.

[TD-er]

@mcspr What is the status of your tests?
Do you have something ready for a PR?

[mcspr]

The issue with the tag message was solved by manual fetch + replace of the current tag, since github checkout uses minimal depth to fetch stuff:
https://github.com/mcspr/ESPEasy/blob/71bf5c57a38007e9cf23ac92e9206fa2f1b9917f/.github/workflows/build.yml#L12
actions/checkout#290
Sorry for ghosting this issue, I did some testing ~2w ago but forgot to follow up ...

Let me rebase this again so I don't have those tagged commits and leftover debug flags.

[tonhuisman]

This issue can be closed, as we have successfully moved to Github Actions 👍