pio pkg exec --package "platformio/tool-esptoolpy" -- esptool version fails in Github Actions

[CocoDico78]

Calling pio pkg exec --package "platformio/tool-esptoolpy" -- esptool version fails in Github Actions runner with a UserSideException: [Errno 13] Permission denied: 'esptool' error.

Calling python -m esptool version works in Github Actions runner, which shows that there is nothing wrong with the underlying esptool tool.

The error appears in Windows and Ubuntu. The error doesn't appear outside of Github Actions runner (ie it "works on my machine").

For repro and logs, you can check the Actions logs of this repo. The failed run shows the error with pio pkg exec ..., the successful run shows the python -m ... call.

[Jason2866]

Wrong way to use the Platformio installed python. See

platform-espressif32/builder/main.py

Line 283 in c6a8c9a

ERASECMD='"$PYTHONEXE" "$OBJCOPY" $ERASEFLAGS erase_flash',

You have to use the Platformio env var for PYTHON to get the path and call

[CocoDico78]

Maybe I misunderstood, but the issue it that

pio pkg exec --package "platformio/tool-esptoolpy" -- esptool --chip esp32 merge_bin --output output.bin 0x1000 test1.bin 0x2000 test2.bin fails in Github Actions runner

I only include the python -m example to show that there is nothing wrong with the underlying esptool command or its parameters. I know it's not the same Python, I am not trying to "use the Platformio installed python".

[Jason2866]

Here is an example to use esptool.py to merge binaries
https://github.com/letscontrolit/ESPEasy/blob/mega/tools/pio/post_esp32.py

[CocoDico78]

I know how to merge binaries using esptool. The issue is that the command fails in a Github Actions runner when called through the pio pkg exec paradigm, see this Actions log.

To prove that it's not an issue with the underlying merge_bin command, I also provide a reference Actions log that works, through a non-platformio way.

The repo I'm linking is specifically made as a minimum repo that reproduces the issue. It's not doing anything except calling merge_bin in two different ways and showing that it fails with pio pkg exec.

[Jason2866]

The snippet shown does work in github actions too.

[CocoDico78]

The snippet shown doesn't use pio pkg exec; it is using esptool.main to call into esptool. It's similar to what I demonstrate here, so I'm not suprised it works.

When using pio pkg exec, it fails in Github Actions however, as demonstrated here.

[Jason2866]

I guess pio calls not the correct Platformio. Try with pinned path

[CocoDico78]

I guess pio calls not the correct Platformio

What is the correct platformio? The Github Actions runs pip install --upgrade platformio and then calls pio. What other platformio is there that it could be mistaken?

Try with pinned path

Pinned path for what? For pio? Also I reproduced the issue with the much simpler esptool version instead of esptool merge_bin [...]. This is not a merge_bin issue but as you say a wiring issue between pio and its tools.

[Jason2866]

Yes, path for calling pio You have probably two Platformio versions installed. The original and the one in the venv
Platformio is generating an venv during installation. Just pio will execute not the needed one from the venv
Which brings me back to the script which makes sure to use the correct version by using the set env variables. If this variables are not used it needs taking care to call the correct one.

[CocoDico78]

You have probably two Platformio versions installed

So you're saying that there is an issue with the pip install --upgrade platformio command itself? As you can see in the workflow file, I don't explicitly install two versions of platformio.

Or is it by design from platformio that there are always two versions that get installed?

[Jason2866]

Yes, by design Platformio installs an venv. That's not an issue from the command. User has to take care to use the "correct" one from the venv
What is your reason for installing the later needed tool manually?
The platform is taking care to install everything needed automatically.

[CocoDico78]

And by design pio will point to the wrong one?

[CocoDico78]

What is your reason for installing the later needed tool manually?

That is a good question. In my automated build, I use pio for the heavy-lifting and later I need to invoke esptool commands. Since I already have pio installed and "platformio/tool-esptoolpy" is already installed, I figured it'd be better to call into pio pkg exec --package "platformio/tool-esptoolpy" -- esptool instead of installing esptool again with python directly. It turns out it doesn't work, as this issue shows.

So I don't need to install the tool manually. I just need to call into it when it's already installed. Which doesn't work (in GH runner).

[Jason2866]

So I don't need to install the tool manually. I just need to call into it when it's already installed. Which doesn't work (in GH runner).

That looks like my statement is correct. There is no path set and so just calling esptool.py fails, since not found.
Solution find the path and store "somewhere" to call it later. Easier to do all needed post compile steps in an Platformio script. You can call easily all tools you need.
You can check via an env var if the compile run happens in github actions and do different than in "normal" builds
Example https://github.com/arendst/Tasmota/blob/ccd76b26ed8c309b3a413b39535d5309bb1abc9e/pio-tools/post_esp32.py#L51

[CocoDico78]

so just calling esptool.py fails, since not found.

I don't call esptool.py, do I? I call into a tool using pio pkg. There is something I don't see?

Solution find the path and store "somewhere" to call it later

I don't follow. What is the point of this tool "platformio/tool-esptoolpy" if it doesn't run as standalone? What else should I run in my .yaml file to wire it up?

Easier to do all needed post compile steps in an Platformio script

It's easier to do pip install esptool and call python -m esptool if you ask me. Don't want to be messing with Python. But that's opinion-based :)

[Jason2866]

Anyways looks like no issue from Platformio here.

[CocoDico78]

looks like no issue from Platformio here.

There is the issue that the "platformio/tool-esptoolpy" package doesn't work in GH Actions. But as you say that's not a general platformio issue (pio pkg exec works just fine in general), that's an issue with the "platformio/tool-esptoolpy" package. Is there a better place for me than this repo to open an issue?

[Jason2866]

I wouldn't say there is an issue. Call the command with path and it will work.

[CocoDico78]

Sure, I've added the full path so now the full command is /opt/hostedtoolcache/Python/3.12.7/x64/bin/pio pkg exec --package "platformio/tool-esptoolpy" -- esptool version (on Ubuntu).

The new GH logs show the same error.

If that's not the correct path, then what is the correct path? And why would pio get installed in the wrong path?

[Jason2866]

This is not the path of the venv!
/opt/hostedtoolcache/Python/3.12.7/x64/bin/pio
pio is not installed in wrong path. It is installed in the venv. You find it ..../.platformio/packages/ in a folder called penv or similar (don't know out of head)

[CocoDico78]

As you may notice in my repro .yaml file, I was using the Python Package Manager installation method. From the docs, it says that it doesn't use a venv. So that location I'm linking is really the only one with platformio. In the ~/.platformio folder, there is nothing else than appstate.json and a packages subfolder.

But here we go, I changed the installation method to the Install Script method which looks like it's installing a venv. I changed the invocation line to ~/.platformio/penv/bin/pio pkg exec --package "platformio/tool-esptoolpy" -- esptool version (see workflow line here) and the error is the same Permission denied: 'esptool' (see GH logs here).

[CocoDico78]

Hey @Jason2866 and thanks for your involvement yesterday, I feel like we're now touching on the heart of the issue and I've tried your suggestion to

Call the command with path and it will work

with the venv but it still doesn't work. Let me know if you need any more evidence of the issue.