Unable to upload or repackage the upstream package #3776

mcspr · 2020-12-24T22:56:50Z

Configuration

Operating system:

Windows 10 20H2

PlatformIO Version (platformio --version):

PlatformIO Core, version 5.0.4b1`

Description of problem

As previously mentioned in the #3612

ESP8266 Core publishes current development version of the toolchain here:
https://github.com/earlephilhower/esp-quick-toolchain

I was trying to upload it to the registry, however this shows up:

> C:\Users\maxim\.platformio\penv\Scripts\pio.exe package publish .\x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz
Error: Traceback (most recent call last):
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\package.py", line 87, in package_publish
    archive_path = p.pack()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\pack.py", line 120, in pack
    with FileUnpacker(src) as fu:
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 152, in __enter__
    self._archiver = self._init_archiver()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 144, in _init_archiver
    with open(self.path, "rb") as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz'

============================================================

As mentioned in the previous issue, I did the pio package pack after this failed, which created another tarball in the same directory:

> C:\Users\maxim\.platformio\penv\Scripts\pio.exe package pack .\x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz
Wrote a tarball to "C:\Users\maxim\xtensa\toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz"

However, the contents are no longer the same:

/m/c/U/m/xtensa> tar vtf x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz | grep g++
hrwxr-xr-x 1000/1000         0 2020-12-23 08:10 xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ link to xtensa-lx106-elf/bin/xtensa-lx106-elf-c++
-rw-r--r-- 1000/1000   1331843 2020-12-23 08:10 xtensa-lx106-elf/share/man/man1/xtensa-lx106-elf-g++.1
/m/c/U/m/xtensa> tar vtf toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz | grep g++
hrw-rw-rw- 0/0               0 2020-12-23 08:10 bin/xtensa-lx106-elf-g++ link to bin/xtensa-lx106-elf-c++
-rw-rw-rw- 0/0         1331843 2020-12-23 08:10 share/man/man1/xtensa-lx106-elf-g++.1

Files are defaulted to rw permissions and UID/GID 0, executable bit lost as well.
edit: although, UID & GID 0 may still be ok i.e. those should not be used anyway without 'numeric owner' setting (?)

So, the issues are:

permissions are lost when unpacking and then packing back when using Windows-based PIO installation
previously I was able to upload and use the version 5.100200.200918, but that was done using Linux installation
it is impossible to upload the original tarball, despite it being compatible with standalone installation method
mcspr/toolchain-xtensa linux build is broken, but I'd assume it will be fixed with re-upload using a different OS (...publishing needs unpublish first, which is not checked before the upload, but only on moderation step :/)

cc @Jason2866 via esp8266/Arduino#7792 (comment)

The text was updated successfully, but these errors were encountered:

mcspr · 2020-12-25T02:53:19Z

Having just tried via WSL, where

original x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz
pio-package-pack'ed toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz

~/xtensa> pio --version
PlatformIO Core, version 5.0.4b1

~/xtensa> pio package publish toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz
Error: Traceback (most recent call last):
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/commands/__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/commands/package.py", line 89, in package_publish
    archive_path = p.pack()
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/package/pack.py", line 120, in pack
    with FileUnpacker(src) as fu:
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/package/unpack.py", line 152, in __enter__
    self._archiver = self._init_archiver()
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/package/unpack.py", line 144, in _init_archiver
    with open(self.path, "rb") as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz'

~/xtensa> pio package publish (readlink -f toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz)
The package has been accepted for moderation. We will notify you on email when process it.

 ~/xtensa> pio package publish (readlink -f x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz)
The package has been accepted for moderation. We will notify you on email when process it.

Surprisingly, using the full path to the archive made this work

ivankravets · 2020-12-26T11:15:07Z

publish .\x86_64

What is it /\? Just use a file name if you are in the current directory.

mcspr · 2020-12-26T11:32:15Z

Windows slashes are backwards, autocomplete puts it when TAB'ing the file in the current directory. It is irrelevant, you can still get the exception without it (c/p was from the 2nd attempt, without the dot-slash)

Jason2866 · 2020-12-26T11:35:35Z

@ivankravets okay about the /\ but this is NOT the main issue here.
If you use Windows to do a package from a Linux tar ball the file permissions get lost.
See the linked issue to Arduino ESP8266 -> New Gcc toolchain packages for the actual Arduino ESP8266 stage

mcspr · 2020-12-26T11:37:01Z

I've uploaded correct tarballs already, trying to upload the one from the bintray:

PS C:\Users\maxim\xtensa> dir

    Directory: C:\Users\maxim\xtensa

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          26.12.2020    12:06       74117637 d10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz

PS C:\Users\maxim\xtensa> C:\Users\maxim\.platformio\penv\Scripts\pio.exe package publish d10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz
Error: Traceback (most recent call last):
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\package.py", line 87, in package_publish
    archive_path = p.pack()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\pack.py", line 120, in pack
    with FileUnpacker(src) as fu:
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 152, in __enter__
    self._archiver = self._init_archiver()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 144, in _init_archiver
    with open(self.path, "rb") as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'd10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz'

ivankravets · 2020-12-26T12:07:35Z

Could you help to investigate this issue? The code is here

Thanks!

mcspr · 2020-12-26T12:38:33Z

A quick summary:

adf9ba2 introduced fs.cd(tmp_dir) while running pio package publish command, it only works with full paths (unless copy happens, somehow, but I don't see it anywhere though)
tests do not cover PackagePacker case for archives, only directories
for this issue specifically, packer can preserve [filepath, TarInfo] from the original archive and utilize TarFile::add(..., filter=map_tar_info) to replace TarInfo with the orginal one

ivankravets · 2020-12-26T13:46:25Z

@Jason2866 thanks, we have reproduced this issue! Downgrading issue to bug.

…ns are not preserved // Resolve // #3776

ivankravets · 2020-12-28T11:08:32Z

Thanks! Fixed, you can publish tar.gz directly from Windows.

mcspr · 2020-12-28T17:27:31Z

@ivankravets Thanks!

Should the pio package pack have a warning or just fail on archives?
btw I noticed that the owner inside of d10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz is pioapp/pioapp, does the remote API repackage yet again? (or I again misunderstood the package flow, sry but I have not really followed the code too deep)

ivankravets · 2020-12-28T17:40:55Z

Should the pio package pack have a warning or just fail on archives?

What do you mean?

is pioapp/pioapp, does the remote API repackage yet again?

I downloaded the package mentioned above and the permissions are good. Yes, we repack EVERY package on the server side and remove any trash from it (pictures, movies, etc). People can control package contents using "package.json" and "export" field as described in docs.

mcspr · 2020-12-28T17:44:56Z

What do you mean?

The other issue after my tests - I was able to do pio package pack $archive on the upstream .tar.gz effectively breaking it. I meant, should it notify the user (or just straight do nothing) about the possible breakage of permissions when running FS without an ability to store unix metadata stuff?

ivankravets · 2020-12-28T17:48:42Z

Yes, we had the same thoughts to show warning message or raise an error. It's not our issue and we can't fix it. Windows does not support symlinks, and other UNIX FS features.

We will raise an exception and ask user to extra archive manually before running "pack".

ivankravets · 2020-12-28T18:13:00Z

Thanks, resolved!

ivankravets closed this as completed Dec 26, 2020

ivankravets reopened this Dec 26, 2020

ivankravets added help wanted package management labels Dec 26, 2020

ivankravets added bug and removed help wanted labels Dec 26, 2020

ivankravets added this to the 5.0.4 milestone Dec 26, 2020

ivankravets self-assigned this Dec 26, 2020

ivankravets added a commit that referenced this issue Dec 28, 2020

Fixed an issue with package publishing on Windows when Unix permissio…

c623a6a

…ns are not preserved // Resolve // #3776

ivankravets closed this as completed Dec 28, 2020

ivankravets reopened this Dec 28, 2020

ivankravets closed this as completed in 4c8c261 Dec 28, 2020

ivankravets mentioned this issue Dec 30, 2020

platformio 5.0.4 Homebrew/homebrew-core#68014

Closed

5 tasks

mcspr mentioned this issue Jan 30, 2021

Package publish rejects the package #3828

Closed

This was referenced Mar 18, 2021

Bump platformio from 5.0.1 to 5.1.1 Alex9779/esphome#28

Closed

Bump platformio from 4.3.4 to 5.1.1 arvidj01/esphome#29

Closed

This was referenced Mar 18, 2021

Bump platformio from 4.3.4 to 5.1.1 AndreaRigoni/esphome#26

Closed

Bump platformio from 5.0.1 to 5.1.1 josla972/esphome#15

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to upload or repackage the upstream package #3776

Unable to upload or repackage the upstream package #3776

mcspr commented Dec 24, 2020 •

edited

Loading

mcspr commented Dec 25, 2020

ivankravets commented Dec 26, 2020

mcspr commented Dec 26, 2020 •

edited

Loading

Jason2866 commented Dec 26, 2020

mcspr commented Dec 26, 2020 •

edited

Loading

ivankravets commented Dec 26, 2020

mcspr commented Dec 26, 2020

ivankravets commented Dec 26, 2020

ivankravets commented Dec 28, 2020

mcspr commented Dec 28, 2020

ivankravets commented Dec 28, 2020

mcspr commented Dec 28, 2020 •

edited

Loading

ivankravets commented Dec 28, 2020

ivankravets commented Dec 28, 2020

Unable to upload or repackage the upstream package #3776

Unable to upload or repackage the upstream package #3776

Comments

mcspr commented Dec 24, 2020 • edited Loading

Configuration

Description of problem

mcspr commented Dec 25, 2020

ivankravets commented Dec 26, 2020

mcspr commented Dec 26, 2020 • edited Loading

Jason2866 commented Dec 26, 2020

mcspr commented Dec 26, 2020 • edited Loading

ivankravets commented Dec 26, 2020

mcspr commented Dec 26, 2020

ivankravets commented Dec 26, 2020

ivankravets commented Dec 28, 2020

mcspr commented Dec 28, 2020

ivankravets commented Dec 28, 2020

mcspr commented Dec 28, 2020 • edited Loading

ivankravets commented Dec 28, 2020

ivankravets commented Dec 28, 2020

mcspr commented Dec 24, 2020 •

edited

Loading

mcspr commented Dec 26, 2020 •

edited

Loading

mcspr commented Dec 26, 2020 •

edited

Loading

mcspr commented Dec 28, 2020 •

edited

Loading