update_installed_extensions.sh will ALWAYS fail with the redhat.java (and possibly others) extensions

[Frontear]

Hi! I've found a pretty nasty (bug? ms issue? extension issue?) thing that I would like to raise as an issue, in case someone else runs into the same mind twisting nonsense.

A few months back I wrote a relatively simple script called code2nix, which was intended to convert my vscode extensions -> nix expressions. It has since worked flawlessly until yesterday, when I tried it and it would consistently fail when pulling the latest versions but would pass when pulling whatever I had currently installed.

Strange! So I went to debugging and attempting tests. What I discovered was honestly mind boggling. Every single time the hash of certain extensions kept randomly changing. Furthermore, there was indeed one singular extension that would consistently fail when pulling its latest. Which one? redhat.java. After realizing this I quickly took to figuring out what the hell was going on, and I discovered that while both were internally VERSIONED the same (1.33.2024072008 at the time of writing), the file content was subtly different.

Here's a simple shell snippet to see what I mean:

$ nix-prefetch-url --print-path "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/java/1.33.2024072008/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage"
0java0yqb6yrycsszlv3vj2hapk5xlx3lr02airrkx09cdlwdrw1
/nix/store/gbmx94y50k4fd5iazrizwm8w4valkflh-Microsoft.VisualStudio.Services.VSIXPackage

However:

$ nix-prefetch-url --print-path "https://redhat.gallery.vsassets.io/_apis/public/gallery/publisher/redhat/extension/java/latest/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage"
1bf97qy9dvc0d37ligkkw6jh1406awb7mdrzamr5gy9q40jmf2bd
/nix/store/zjikf6maii5b21qiyxh609dqa7h1xhqj-Microsoft.VisualStudio.Services.VSIXPackage

A completely different narHash and store path!? Now of course you must be thinking, well obviously these must be completely different versions of the extension, right? Well...

$ nix run nixpkgs#unzip -- -qc "/nix/store/gbmx94y50k4fd5iazrizwm8w4valkflh-Microsoft.VisualStudio.Services.VSIXPackage" "extension/package.json" | nix run nixpkgs#jq -- -r '.version'
1.33.2024072008
$ nix run nixpkgs#unzip -- -qc "/nix/store/zjikf6maii5b21qiyxh609dqa7h1xhqj-Microsoft.VisualStudio.Services.VSIXPackage" "extension/package.json" | nix run nixpkgs#jq -- -r '.version'
1.33.2024072008

For some reason, these files are versioned the exact same, but actually contain some kind of difference. I extracted the files using unzip and reviewed the file structure with diff and found:

diff --unified --recursive --text latest/[Content_Types].xml version/[Content_Types].xml
--- latest/[Content_Types].xml  2024-07-20 08:07:48.000000000 +0000
+++ version/[Content_Types].xml 2024-07-20 08:11:08.000000000 +0000
@@ -1,2 +1,2 @@
 <?xml version="1.0" encoding="utf-8"?>
-<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension=".json" ContentType="application/json"/><Default Extension=".vsixmanifest" ContentType="text/xml"/><Default Extension=".ts" ContentType="video/mp2t"/><Default Extension=".md" ContentType="text/markdown"/><Default Extension=".txt" ContentType="text/plain"/><Default Extension=".css" ContentType="text/css"/><Default Extension=".jar" ContentType="application/java-archive"/><Default Extension=".png" ContentType="image/png"/><Default Extension=".xml" ContentType="application/xml"/><Default Extension=".js" ContentType="application/javascript"/><Default Extension=".ini" ContentType="text/plain"/><Default Extension=".py" ContentType="application/octet-stream"/><Default Extension=".bat" ContentType="application/x-msdownload"/><Default Extension=".properties" ContentType="application/octet-stream"/><Default Extension=".dat" ContentType="application/octet-stream"/><Default Extension=".ja" ContentType="application/octet-stream"/><Default Extension=".lib" ContentType="application/octet-stream"/><Default Extension=".cfg" ContentType="application/octet-stream"/><Default Extension=".src" ContentType="application/x-wais-source"/><Default Extension=".bfc" ContentType="application/octet-stream"/><Default Extension=".sym" ContentType="application/octet-stream"/><Default Extension=".h" ContentType="text/x-c"/><Default Extension=".dll" ContentType="application/octet-stream"/><Default Extension=".exe" ContentType="application/octet-stream"/><Default Extension=".policy" ContentType="application/octet-stream"/><Default Extension=".certs" ContentType="application/octet-stream"/><Default Extension=".jfc" ContentType="application/octet-stream"/><Default Extension=".security" ContentType="application/octet-stream"/><Default Extension=".template" ContentType="application/octet-stream"/><Default Extension=".access" ContentType="application/octet-stream"/></Types>
+<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension=".json" ContentType="application/json"/><Default Extension=".vsixmanifest" ContentType="text/xml"/><Default Extension=".ts" ContentType="video/mp2t"/><Default Extension=".md" ContentType="text/markdown"/><Default Extension=".txt" ContentType="text/plain"/><Default Extension=".css" ContentType="text/css"/><Default Extension=".jar" ContentType="application/java-archive"/><Default Extension=".png" ContentType="image/png"/><Default Extension=".xml" ContentType="application/xml"/><Default Extension=".js" ContentType="application/javascript"/><Default Extension=".ini" ContentType="text/plain"/><Default Extension=".py" ContentType="application/octet-stream"/><Default Extension=".bat" ContentType="application/x-msdownload"/></Types>
Only in latest/extension: jre
diff --unified --recursive --text latest/extension.vsixmanifest version/extension.vsixmanifest
--- latest/extension.vsixmanifest       2024-07-20 08:07:48.000000000 +0000
+++ version/extension.vsixmanifest      2024-07-20 08:11:08.000000000 +0000
@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="utf-8"?>
        <PackageManifest Version="2.0.0" xmlns="http://schemas.microsoft.com/developer/vsx-schema/2011" xmlns:d="http://schemas.microsoft.com/developer/vsx-schema-design/2011">
                <Metadata>
-                       <Identity Language="en-US" Id="java" Version="1.33.2024072008" Publisher="redhat" TargetPlatform="win32-x64"/>
+                       <Identity Language="en-US" Id="java" Version="1.33.2024072008" Publisher="redhat" />
                        <DisplayName>Language Support for Java(TM) by Red Hat</DisplayName>
                        <Description xml:space="preserve">Java Linting, Intellisense, formatting, refactoring, Maven/Gradle support and more...</Description>
                        <Tags>multi-root ready,keybindings,json,java,__ext_java,__ext_class,java-properties,Java Properties,__ext_properties,gradle-kotlin-dsl,Gradle Kotlin DSL,__ext_gradlekts,linters</Tags>

Seems the difference is in the extension.vsixmanifest and [Content_Types].xml. I'm not quite sure who to pin the blame for this since this is an issue that I can only so far find with this specific extension. Maybe its a combination of both.

So why does this all matter? This matters because https://github.com/NixOS/nixpkgs/blob/71770c17e76268cb001cf03ad670a01db8d00ad6/pkgs/applications/editors/vscode/extensions/update_installed_exts.sh will occasionally fail if given extensions like this. Actually this was the second script I attempted to try after my own app failed me, thinking that maybe I had actually missed a bug, but no, it turns out the root issue was impure nonsense from upstream. This is an issue that can easily propagate and can lead to a lot of confusing issues if not known about.

The personal fix I am planning to implement is to pull the latest package, pull the version, then re-download the versioned package and use that for hashing. Super inefficient but its unfortunately necessary in this context. I don't really know if a similar approach is feasible for nixpkgs, but I figure I should both share my findings and my take on a "solution" for others who may struggle with the same behaviour.

If anyone feels like having an evening laugh, feel free to take a look at the commit history here, noting especially how the hashes will randomly change when pulling certain versioned things vs. certain latest things. My personal favorite has to be this commit where the hashes completely swap despite no logical reason to.

[Frontear]

This issue still exists, even with the aforementioned "fix" of re-downloading the package. I suspect upstream repacks the extensions in some cases, which leads to a different FOD. May or may not be related to #347212, I don't actually know the full internals of extension FODs on nixpkgs so this may not be true.