Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

temurin-11-jre and temurin-17-jdk Debian package aren't served properly on JFrog #1006

Closed
hdurand-work opened this issue Jan 26, 2024 · 22 comments
Labels
bug Something isn't working help wanted Extra attention is needed Waiting on OP

Comments

@hdurand-work
Copy link

hdurand-work commented Jan 26, 2024

Hello,

Same as #1002 (but it was closed)

Since yesterday packages.adoptium.net stopped serving some of the temurin's Debian package.

debmirror's log:

Errors:
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-11/temurin-11-jre_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
Failed to download files (3 errors)

Our firewall doesn't seem to see any problem:

image

After digging into it, we discovered that the packages aren't visible on JFrog's web UI even though it seems to "know" that they exist

image

image

Alpine in comparison:

image

Debian 11 JRE in comparison:
image

@karianna
Copy link
Contributor

adoptium/installer#810 to resolve this

@steelhead31
Copy link

I also believe these packages should now be being served properly.

@karianna
Copy link
Contributor

@hdurand-artic Can you try again and let us know?

@sxa
Copy link
Member

sxa commented Jan 26, 2024

This is odd. I just performed a quick new test on Debian 12 (bookworm) on x64 and 11-jre installed ok but 17-jdk didn't.

Err:1 https://packages.adoptium.net/artifactory/deb bookworm/main amd64 temurin-17-jdk amd64 17.0.10.0.0+7
  Error reading from server. Remote end closed connection [IP: 151.101.131.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb  Error reading from server. Remote end closed connection [IP: 151.101.131.42 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
root@f76153b94ba7:/# wget https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb
--2024-01-26 23:28:26--  https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb
Resolving packages.adoptium.net (packages.adoptium.net)... 151.101.67.42, 151.101.195.42, 151.101.3.42, ...
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.67.42|:443... connected.
HTTP request sent, awaiting response... 200 OK

I tried the install several times with the same results, then paused for a minute or so before trying the wget on the URL, which worked, and then after that the apt install worked. It does feel somehow cache/proxy related.

EDIT: Tried in a new docker container after the above and it worked ok straight away with temurin-17-jdk - maybe my wget did some magic :-)

@hdurand-work
Copy link
Author

hdurand-work commented Jan 29, 2024

@karianna Some errors are still here, including on new packages:

Errors:
 Download of pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-11/temurin-11-jre_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-17/temurin-17-jre_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-8/temurin-8-jre_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Failed to download files (8 errors)!

Some of them are indeed broken in JFrog - for example:

image

image

but, weirdly enough, not all of them: - for example:

image

image

@steelhead31
Copy link

I've checked all of these artifacts ( and re-uploaded them, and checked again! ), and it appears all are present and correct, there may be an issue with the jfrog web UI, when clicking on the location page. but for me at least, it then redirects to the correct artifact when browsing, e.g...  https://packages.adoptium.net/ui/repos/tree/General/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.1.0.0+12_amd64.deb   .. Im going to log the issue with the webui with jfrog.. but I've also tested that apt install works for JDKs & JREs for all version and installs the current/latest versions..

@hdurand-work
Copy link
Author

hdurand-work commented Jan 29, 2024

@steelhead31

I am not sure it's working... it doesn't seem to error out on download, but it does say that there were timeouts... which is weird:

Files to download: 568 MiB
[  0%] Getting: pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb... [  0%] Getting: pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb... [  0%] Getting: pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb... [  0%] Getting: pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb... Downloaded 25 kiB in 25s at 1.03 kiB/s.
Everything OK. Moving meta files ...
Cleanup mirror.
All done.
Errors:
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
Failed to download files (4 errors)!

@steelhead31
Copy link

@hdurand-artic Are you using a mirror repo/proxy or anything of that nature?, also would you mind sharing your repository configurations, and specifically what/how you are downloading the jdks ? .. the read errors appear to be related to debmirror..

@hdurand-work
Copy link
Author

@steelhead31 So, we're using debmirror to mirror every adoptium debian packages (amd64 / bullseyes-buster) - and serve them back ourselves internally.

I don't think there's a proxy between our server and your server.

(I'm personally not super familiar with debmirror)

Everything worked before the original issue happened so I really don't think it's on our end.

@steelhead31
Copy link

steelhead31 commented Jan 29, 2024

@hdurand-artic  I think the cache on the debmirror may need updating, as we had to re-upload some packages following an initial error... is it possible for you to try with the debmirror disabled?, so you are downloading directly.

@hdurand-work
Copy link
Author

is it possible for you to try with the debmirror disabled?

It really isn't no, our machines don't have external access if they don't have to

@steelhead31
Copy link

I've reached out to jfrog to check everything on their end, especially the fact that the web-ui doesn't display any packages, though it does list 13 available.

@hdurand-work
Copy link
Author

@steelhead31 So, after digging into the debmirror documentation, I found out about "precleanup" after running it once I got those errors:

Errors:
 Download of pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb failed: 500 read timeout
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Failed to download files (4 errors)!

but those errors don't appear again if I run it again with or without cleanup... I guess something is still wrong on your end?

@steelhead31
Copy link

Yes, it appears there may be an issue on our hosted jfrog server, I've raised an issue with them to investigate, but it looks related to metadata/indexes on the artifactory server itself.  The packages are visible ( and installable via other methods ) . But I think I've exhausted everything I can do.. I'll update this issue when I get an update

@hdurand-work
Copy link
Author

@steelhead31 Alright, thank you!

@pandeyv-del
Copy link

pandeyv-del commented Feb 2, 2024

Adding more details on this.

case1: if i add repo details on the CentOS/fedora on the server then there is no error and system will able to download the package.
cat <<EOF > /etc/yum.repos.d/adoptium.repo [Adoptium] name=Adoptium baseurl=https://packages.adoptium.net/artifactory/rpm/${DISTRIBUTION_NAME:-$(. /etc/os-release; echo $ID)}/\$releasever/\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.adoptium.net/artifactory/api/gpg/key/public EOF

Cas2: if i add these in some kind of script(in my case i was doing on json for packer) and add the repo detail then first it will start throwing error around escape character echo $ID)}/\$releasever/\$basearch if we remove escape character then it will not throw any error but system will not be able to download the package saying repodata xml is not valid to resolve this i have escaped the escape character also like this
baseurl=https://packages.adoptium.net/artifactory/rpm/${DISTRIBUTION_NAME:-$(. /etc/os-release; echo $ID)}/\\$releasever/\\$basearch

After this fix packer json also valid and system was able to download the package.

@steelhead31
Copy link

@pandeyv-del thanks for that information, thats useful to know...

@hdurand-artic Apologies Im still waiting on an update from Jfrog for this issue, I've chased them again this morning.  Can you have a look at the comment above and see if something similar is appropriate for your mirror scripting / environment?

@hdurand-work
Copy link
Author

@steelhead31 Hello, do you have any news ?

By the way, here's the debmirror output with debug/verbose enabled (I had to dug a bit recently to understand how debmirror works - after an unrelated issue - so I was able to enable that)

Get Translation files ...
Get DEP-11 metadata files ...
Get command-not-found metadata files ...
Files to download: 568 MiB
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb => [  0%] Getting: pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb...     #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb ==> 200 OK (4s)
200 OK
read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb => [  0%] Getting: pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb...     #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb ==> 500 read timeout (3s)
500 read timeout
Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb => [  0%] Getting: pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb...     #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb ==> 500 read timeout (3s)
500 read timeout
Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb => [  0%] Getting: pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb...         #** GET https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb ==> 500 read timeout (3s)
500 read timeout
Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Downloaded 38 kiB in 20s at 1.93 kiB/s.
Everything OK. Moving meta files ...
Moving dists/bullseye/InRelease
Moving dists/bullseye/Release
Moving dists/bullseye/main/binary-all/Packages.gz
Moving dists/bullseye/main/binary-amd64/Packages.gz
Moving dists/bullseye/Release.gpg
Moving dists/buster/InRelease
Moving dists/buster/Release
Moving dists/buster/main/binary-all/Packages.gz
Moving dists/buster/main/binary-amd64/Packages.gz
Moving dists/buster/Release.gpg
Moving dists/bookworm/InRelease
Moving dists/bookworm/Release
Moving dists/bookworm/main/binary-all/Packages.gz
Moving dists/bookworm/main/binary-amd64/Packages.gz
Moving dists/bookworm/Release.gpg
Cleanup mirror.
deleting pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb-4017329
All done.
Errors:
 read timeout at /usr/share/perl5/Net/HTTP/Methods.pm line 249. at /usr/bin/debmirror line 1828.
 Download of pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0+7_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13_amd64.deb failed: 500 read timeout
 Download of pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb failed: 500 read timeout
Failed to download files (4 errors)!

@hdurand-work
Copy link
Author

hdurand-work commented Feb 14, 2024

So it seems to work now.

I also just noticed that URLs had double "/" "https://packages.adoptium.net/artifactory/deb//pool/main/t/temurin-8/temurin-8-jdk_8.0.402.0.0+6_amd64.deb" , after solving this.

It seems to be working consistently... so problem solved? Maybe? https://packages.adoptium.net/ui/packages/deb:%2F%2Ftemurin-11-jdk?name=temurin-11&type=packages still broken on JFrog though.

@steelhead31
Copy link

Thanks for the update, Im still awaiting a response from jfrog... ,so Im guessing the caching issue with broken URLs is now working properly... as far as the packages, Im pretty much stuck until jfrog take a look.. as they manage the repository, and the only option left to me, is to rebuild the repository from scratch.. which I'd like to avoid unless its absolutely necessary.

Copy link

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable.
It will be closed soon unless the stale label is removed by a committer, or a new comment is made.

@github-actions github-actions bot added the stale label May 15, 2024
@karianna karianna removed the stale label May 15, 2024
@steelhead31
Copy link

I think these issues have now been resolved. Certainly the most recent release packages work correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed Waiting on OP
Projects
None yet
Development

No branches or pull requests

5 participants