gzip from Ubuntu Eoan doesn't execute

[iBug]

• Your Windows build number: 1903 (18362.295)

• What you're doing and what's happening:

  wsl@MSI-GS73:~ $ gzip
  -bash: /bin/gzip: cannot execute binary file: Exec format error
  126|wsl@MSI-GS73:~ $ apt list gzip
  Listing... Done
  gzip/eoan,now 1.10-0ubuntu3 amd64 [installed]
  wsl@MSI-GS73:~ $
  

• What's wrong / what should be happening instead:

  gzip from Ubuntu Eoan gives the above unexpected error. The same binary behaves perfectly well on a native Linux box (tested on Ubuntu Bionic / Disco and Debian Stretch / Buster).

• Strace of the failing command, if applicable: No strace because the binary doesn't run at all :(

• For WSL launch issues, please collect detailed logs.

See our contributing instructions for assistance.

[Biswa96]

I can reproduce your issue in latest Windows 10 Pro insider build 18970. Here are some points that I found:

• Ubuntu Disco and lower: file $(which gzip) command shows ELF 64-bit LSB shared object.
• Ubuntu Eoan: file $(which gzip) command shows ELF 64-bit LSB pie executable.
• KD output: LX: (FFFF800368A7C2C0, FFFF800367A04080) (-bash) LxpElfInfoParse: ElfSizeAdd
• Conclusion: The bug is in the ELF parsing function in LxCore driver.
• Workaround: Use WSL2.

[therealkenc]

The interesting thing is they are all PIE (f.e. /bin/ls), and yet the whole Ubuntu edifice doesn't fall down. I took a quick look with readelf(1) and there isn't anything glaring, but isn't saying much.

Reasonable enough WSL1 work around would be to install your own /usr/local/bin/gzip I guess, although we can't know what other executables other than gzip might be problematic.

[skitt]

• Ubuntu Disco and lower: file $(which gzip) command shows ELF 64-bit LSB shared object.
• Ubuntu Eoan: file $(which gzip) command shows ELF 64-bit LSB pie executable.

The difference here isn’t related to a change in gzip, but to a change in file. Ubuntu binaries have been built with PIE for a while; but it’s only recently (coreutils 5.33) that file has started identifying them as PIE executables rather than shared objects.

[chutzimir]

For what it's worth, file in eoan produces this output for the two versions of gzip. It seems that some comments in the file are moved around:

$ dpkg -x ./gzip_1.9-3_amd64.deb 1.9-good
$ dpkg -x ./gzip_1.10-0ubuntu3_amd64.deb 1.10-bad
$ file */bin/gzip
1.10-bad/bin/gzip: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=bc0f5994544c2a469d04c914bf4bf44b4ded6040, for GNU/Linux 3.2.0, stripped
1.9-good/bin/gzip: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=efa859c26eaf8e035efe9a139361e2a60cd17b3e, stripped
$ ./1.9-good/bin/gzip --version | head -n1
gzip 1.9
$ ./1.10-bad/bin/gzip --version
-bash: ./1.10-bad/bin/gzip: cannot execute binary file: Exec format error
$ uname -r
4.4.0-18362-Microsoft

[sirredbeard]

Does this need to be reported upstream to Ubuntu?

[Biswa96]

Nope. Because WSL2 works.

[sirredbeard]

Nope. Because WSL2 works.

Some of us luddites still use WSL1 on our production machines. 😉

I was able to reproduce on Ubuntu Eoan with gzip 1.10.

I cannot reproduce on Debian testing (Pengwin) or unstable which are shipping gzip 1.9-3.

I built gzip 1.10 from source tar from FSF on Debian unstable and Ubuntu Eoan and there was no issue.

[Biswa96]

The issue is in LxCore driver which is part of WSL1 not WSL2, see my first comment, KD output. Hence the issue is from WSL side. PIE is a good security feature. See these two wiki:

• https://wiki.ubuntu.com/Security/Features#pie
• https://wiki.ubuntu.com/SecurityTeam/PIE

[therealkenc]

The problem doesn't appear to be PIC per se, as skitt points out (thanks skitt). Something is weird about the build. It would take a dive to find out what.

WSL1 on our production machines

Production machine with a unreleased Ubuntu build. Okay.

That said, finding problems like this early is a Good Thing, naturally. Just yanking your chain. #3023 is still on the books for what it is worth. Which was also caught "early".

It might be worthwhile for someone motivated to report the problem upstream and see if the Canonical peeps have any idea what changed. Also trying bleeding edge Debian binaries (Bookworm?) might be worthwhile to see if Canonical just inherited the change.

This is a completely legit WSL1 bug, obviously; but in parallel, making gzip play nice like all of (or most of) the other Eoan binaries would also be a Good Thing.

PIE is a good security feature.

It is. On the other hand...

[Biswa96]

It would take a dive to find out what.

If ntoskrnl allows to patch kernel mode driver then it would be possible to take a dive.

[therealkenc]

I meant dive the Ubuntu gzip package. It was working in Disco. Then it wasn't on Eoan. Finding out what changed in the package or the package's dependencies and/or the build.

[sirredbeard]

They updated it from gzip 1.9 to 1.10.

I .diff'd Ubuntu source packages for both and nothing jumped out at me.

I .diff'd the binary packages in a Hex editor and noted some changes in the headers in the gzip binary.

binary_folders_for_diff.zip

source_folders_for_diff.zip

[sirredbeard]

Reported to Ubuntu here.

[sirredbeard]

From upstream:

I tried comparing the binaries with diffoscope, and one difference which jumped out at me is the .plt.sec section which is present in the Eoan binary. This presumably comes from CET (gcc -fcf-protection). There are other differences, less significant IMO; .plt.sec is a likely candidate for something WSL1 might not support.

[therealkenc]

That section is "new" alright. I blindly stripped the .plt.sec, similar to #3023, but no joy. Not that blindly stripping a section out of ELF binary would have any expectation of working. Now, thing is, all of the new binaries (f.e. /bin/ls) have that .plt.sec section also, and they work just fine. Also compiling a hello.c with -fcf-protection works. Because of course it does.

The only other new section in there (Disco gzip has 27 sections, Eoan has 29) is .note.gnu.property. No joy after stripping that either. Meh. It's not .note.ABI-tag; I stripped that as a WAG right after this issue was posted.

Which is all a long description of things that don't work. Next thing to try would be to grab the apt-source, try to build a broken binary, and then try to guess what makes gzip so special (contrast say /bin/ls). Still an outside chance gzip can be patched, somehow. But we're deep into work-around territory more complicated than just recompiling from source.

[benhillis]

Thanks for filing this issue, I identified where this is blowing up in our elf loader and will submit a fix.

[sirredbeard]

I have updated the bug report filed on gzip in Launchpad accordingly.

[rbalint]

@benhillis could you please backport the fix to non-insider versions?

[rbalint]

@benhillis we already publish 19.10 tarballs for WSL and with this bug users can't really use them with WSL1 :-(

[benhillis]

@rbalint - Yes, I will see about getting this backported.

[BobCu]

Despite the WSL1 elf loader being the actual problem, is "Build gzip from source" the correct workaround for WSL1/eoan until the fix is released?

If so, could the "most correct" procedure for doing this be posted to this thread? (I landed here via a Google search.)

[maxb]

I locally rebuilt the gzip package in eoan, and the problem persists.

@benhillis Are you able to share any details about the nature of the ELF loader bug, which might suggest tweaks to compiler or linker options that might avoid it?

@BobCu The simplest short term workaround might be to just downgrade to the gzip package from Ubuntu disco.

[benhillis]

@maxb - Sure. The issue is that our loader was not correctly handling program headers that would exceed the size of the file itself. The offending program header file size of zero but an offset that was greater than the size of the binary.
Hope that helps.

[BobCu]

@maxb - I got a working build via the sources at https://ftp.gnu.org/gnu/gzip/gzip-1.10.zip (note the suffix - .tar.xz also works).

[therealkenc]

The bizarre/frustrating thing about this one is there is nothing obvious in the gzip 1.10 build (when I looked last month) that would cause a "header file size of zero but an offset that was greater than the size of the binary". You can come up with all kinds of imaginable scenarios where that might happen; up to and including some newly introduced and obscure toolchain bug. But nothing I could find to make gzip special, contrast a (say) a toolchain bug that took down the whole Eoan edifice on WSL. Actually the gzip build is about as bog standard as it gets. And of course my local build worked fine, which sheds no light either. Ugh.

[BobCu]

FWIW, I attempted to build the gzip deb via dpkg-buildpackage but couldn't get it to succeed, no doubt due to my unfamiliarity with debs (I build mainly RPMs and tarballs). However, I doubt rebuilding and reinstalling the deb is the best short-term workaround for this issue.

My gzip is generic (without Ubuntu/Debian patches), and is installed in /usr/local/bin (the default), which precedes /bin in my PATH, permitting the default gzip to remain installed (and updated/tested).

Are there any patches that should be applied to gzip as part of the temporary workaround?

Edit: Decided to go with Disco's gzip (see below), so did sudo make uninstall to remove my local gzip build.

[BobCu]

Given the Ubuntu/Debian patches to gzip, it may be safest to follow @maxb's recommendation to downgrade to Disco's gzip:

1. Grab the package from here: https://packages.ubuntu.com/disco/amd64/gzip/download
2. Install it: sudo dpkg -i ./gzip_1.9-3_amd64.deb
3. Protect it from automatic upgrade: sudo apt-mark hold gzip

When this issue has been resolved:

1. Enable upgrade: sudo apt-mark unhold gzip

[ITWARRIORS-Kilian]

I just encountered the same issue again with ubuntu 22.04 in wsl1.

Temporary solution proposed above did work for me as well.

[Rokiers]

if you install remote:wsl or development in vscode ,conflict will been happend

[socram8888]

I am also affected by this bug again in Ubuntu 22.04. Weirdly enough, the rest of the binaries work fine. Only gzip seems to fail.

[westfly]

I am also affected by this bug again in Ubuntu 22.04. Weirdly enough, the rest of the binaries work fine. Only gzip seems to fail.

use this command may help

echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189))

[socram8888]

Huh what does this do? I mean, I get it inserts a 0x10 byte at offset 0x189, but I'm curious what structure are you patching for it to get fixed.

[zeronycs]

I got the same error under Ubuntu 22.04, gzip 1.10, WSL1.
Solved by upgrading gzip to a newer version, in my case 1.12.

1. Download it from here (with wget / curl): http://ftp.debian.org/debian/pool/main/g/gzip/
2. Install it: sudo dpkg -i ./gzip_1.12-1_amd64.deb

[pokatomnik]

@nsynecs, thank you, fixed right now.
I used this version for WSL Ubuntu 22.04 x86_x64.

[as17501]

@nsynecs it works well,thanks!! my WSL1 version is Ubuntu 22.04 LTS.

wget http://ftp.debian.org/debian/pool/main/g/gzip/gzip_1.12-1_amd64.deb   
sudo dpkg -i ./gzip_1.12-1_arm64.deb

[JieChongHai]

@nsynecs Thank you very much. It worked very well！

[davidfordaus]

@as17501 's solution works for WSL 1, Ubuntu 22.04. eg:

wget http://ftp.debian.org/debian/pool/main/g/gzip/gzip_1.12-1_amd64.deb   
sudo dpkg -i ./gzip_1.12-1_arm64.deb

That being said - I'm surprised this bug to a core piece of Ubuntu software (gzip) used in a distribution somehow made it into production. Am I missing something and there's very little testing of WSL, or is this a separation of concerns issue where "WSL works" from a Microsoft standpoint, but the Ubuntu distribution doesn't; eg it's Canonical's issue. Should we be raising it with Canonical as the issue has been around since 2019. Cheers

[kevinkhill]

@as17501 in your example you wget amd but then dpkg arm

[CarlosNihelton]

The current issue with GZip on Ubuntu is better described in #8219 and it's cause is different from this thread. It's caused by the WSL 1 ELF loader. While there might be ways to fix that distro side, it's quite unpredictable when and which binary will be the next affected by such problem.

Anyone willing to confirm what's been reported in the referred issue can try the following command to patch the gzip binary and clarify whether that's accurate or a new issue arose:

echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189))

[StargazerAlex]

The current issue with GZip on Ubuntu is better described in #8219 and it's cause is different from this thread. It's caused by the WSL 1 ELF loader. While there might be ways to fix that distro side, it's quite unpredictable when and which binary will be the next affected by such problem.

Anyone willing to confirm what's been reported in the referred issue can try the following command to patch the gzip binary and clarify whether that's accurate or a new issue arose:

echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189))

I can confirm that this works! I had the same problem as the people above. With WSL 1 Ubuntu 22.04 installing Visual Studio Code would throw the following error:

$ code .
Updating VS Code Server to version b06ae3b2d2dbfe28bca3134cc6be65935cdfea6a
Removing previous installation...
Installing VS Code Server for x64 (b06ae3b2d2dbfe28bca3134cc6be65935cdfea6a)
Downloading: 100%
@����@@@�������������������: Permission denied
/usr/bin/gzip: 3: ���: not found
/usr/bin/gzip: 4: Syntax error: "(" unexpected
tar: Child returned status 2
tar: Error is not recoverable: exiting now

tar is unable to read /home/alange/.vscode-server/bin/b06ae3b2d2dbfe28bca3134cc6be65935cdfea6a-1658171981.tar.gz. Either the file is corrupt or tar has an issue.
There's a known WSL issue with tar on Ubuntu 19.10.
See workaround in https://github.com/microsoft/vscode-remote-release/issues/1856.
Reload the window to initiate a new server download.

After running your magic line it immediately worked:

$ code .
Updating VS Code Server to version b06ae3b2d2dbfe28bca3134cc6be65935cdfea6a
Removing previous installation...
Installing VS Code Server for x64 (b06ae3b2d2dbfe28bca3134cc6be65935cdfea6a)
Downloading: 100%
Unpacking: 100%
Unpacked 2799 files and folders to /home/<user>/.vscode-server/bin/b06ae3b2d2dbfe28bca3134cc6be65935cdfea6a.

[JohnMilazzo]

echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189))

Fixed it for me as well.

[davidfordaus]

Thanks for the workarounds, though I don't see a core fix to the actual underlying issue I'm afraid. It also seems somewhat bizarre that a change was made to WSL that fundamentally broke a core part of the system, yet somehow that made it through the testing processes and was released to production.

[Dvill777]

Hello, I have problems executing the remote wsl, I will leave the attached error.
note: i also have the problem when using code . from the ubuntu 22.04 lts terminal, this does nothing.

[summeriver13]

Hello, I have problems executing the remote wsl, I will leave the attached error. note: i also have the problem when using code . from the ubuntu 22.04 lts terminal, this does nothing.

I have the same error with you.

I fix it by use echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189)) in WSL2,
than use code .

[escalonn]

I thought the gzip patch fixed my issue on WSL 2, but as it turned out, I was using WSL 1 and didn't realize it! run wsl -l -v to be sure.

[strugglebak]

Given the Ubuntu/Debian patches to gzip, it may be safest to follow @maxb's recommendation to downgrade to Disco's gzip:

1. Grab the package from here: https://packages.ubuntu.com/disco/amd64/gzip/download
2. Install it: sudo dpkg -i ./gzip_1.9-3_amd64.deb
3. Protect it from automatic upgrade: sudo apt-mark hold gzip

When this issue has been resolved:

1. Enable upgrade: sudo apt-mark unhold gzip

gzip_1.12-1_amd64.deb can also resolve this problem

[davidfordaus]

My question is twofold: 1. How was WSL 2 released with such a significant bug present in one of the supported platforms ? 2. Is there an automated testing strategy as per normal DevSecOps processes ? If so, where is this documented

[chalau]

The current issue with GZip on Ubuntu is better described in #8219 and it's cause is different from this thread. It's caused by the WSL 1 ELF loader. While there might be ways to fix that distro side, it's quite unpredictable when and which binary will be the next affected by such problem.

Anyone willing to confirm what's been reported in the referred issue can try the following command to patch the gzip binary and clarify whether that's accurate or a new issue arose:

echo -en '\x10' | sudo dd of=/usr/bin/gzip count=1 bs=1 conv=notrunc seek=$((0x189))

It works!

[Tanzeel-inline]

This command fixed the issue for me: sudo apt-get install gzip.

[alfchao]

I got the same error under Ubuntu 22.04, gzip 1.10, WSL1. Solved by upgrading gzip to a newer version, in my case 1.12.

1. Download it from here (with wget / curl): http://ftp.debian.org/debian/pool/main/g/gzip/
2. Install it: sudo dpkg -i ./gzip_1.12-1_amd64.deb

It's work! yeah! Thanks!!!

[CarlosNihelton]

gzip package version 1.10-4ubuntu4.1 was published with a patch that addresses this particular issue. sudo apt update && sudo apt install gzip is enough to install it.

[AKKSHAAT]

@ScarletFlash
i am still facing this issue is there a solution yet?
also https://packages.ubuntu.com/disco/amd64/gzip/download does not work

[ScarletFlash]

@ScarletFlash i am still facing this issue is there a solution yet? also https://packages.ubuntu.com/disco/amd64/gzip/download does not work

Actually, Idk. I'm using a different OS right now.
But you may ask @maxb — that was his solution.