PackageCloud repos are temporarily unavailable

[michaelklishin]

Team RabbitMQ account on PackageCloud is no longer available

Update from May 28th

PackageCloud have generously unblocked our account for a short period of time (say, a week)
in hoping that we can resolve the root cause. However, we cannot guarantee that it will be the
case, and encourage all users to switch to our own mirrors mentioned in the doc guides.

Original post

Team RabbitMQ's PackageCloud org has been suspended for bureaucratic reasons. I'm afraid we cannot know when the org may be back, if ever. We'd like to continue using PackageCloud as one of the
options but right now, there is no approved way for us to pay for the service. It may take a long
time to resolve.

Please switch to our Cloudsmith mirror for Debian and RPM package repositories.

If and when we have an update on the PackageCloud availability situation, it will be shared in this discussion.

Why Was Not This Pre-announced?

Because the core team was not notified of certain changes that resulted in our PackageCloud account
being deactivated until the last day.

We expected certain changes and learned some lessons from running out of a Cloudsmith traffic quota
in April, both Debian and RPM doc guides have been recommending a mirror that has no traffic quotas and is not subject to the same corporate bureaucracy.

[mvarchdev]

:(

[noonedeadpunk]

The problem we have with cloudsmith, is that package rotation there happens quite unpredictable and too aggressively.

We're installing fixed versions of rabbitmq and Erlang, but it's close to be impossible to keep up with version bumps there, so we end up with always broken gates for stable branches due to it.

Is there any way to improve package lifecycle on cloudsmith then, so it could be really used as alternative?

[noonedeadpunk]

But it's not really possible to go just with latest minor version, since Erlang comparability does change inside minor versions?

So if you have fixed Erlang and rabbitmq on some minor version, at some point rabbitmq might become incompatible with it? At least I catched that couple of times.

Moreover, PackageCloud contained already fully rotated and no longer available major versions of rabbitmq.

Our usecase - we have an open source project that sums to deploy environments using Ansible in a reproducible manner for those, who don't want to deal with docket and might need dedicated bare metal nodes solely for rabbit.
And we also have limited capacity on fixing CI and backporting config changes that are required, when older major versions are no longer available.

It would be perfectly fine if rabbitmq-server was published like for erlang-rpm (except also having deb packages).

But also I didn't know about these novemberain.com mirrors, I will try them out!

[michaelklishin]

We automatically clean up patch releases, entire minors are never removed by the clean-up mechanism I am describing above.

Erlang compatibility changes between minor releases and when there are very serious reasons (such as 9+ CVSS score CVE) to bump it in a patch release.

Our team does not have the resources (nor interest) to produce a zero dependency single Erlang package for Debian. For RPM, we had to do
it back in 2011 because the Erlang RPM packaging situation was catastrophically bad.
We do not want to be packaging Erlang, hosting large Erlang packages for free for everyone
forever, even if the footprint is a few TBs a month.

Like I said, to my knowledge the mirrors on {ppa1,yum1}.novemberain.com (that I have personally set up) store the entire catalog of releases starting with 3.9 or so. So does GitHub (the entire catalog going back to at least 3.5). So does Docker Hub.

For open source projects that include RabbitMQ, I do not see any reason to not use the latest RabbitMQ, and N-1 or (usually, not right now) the latest Erlang release. RabbitMQ is usually
compatible with Erlang releases before they ship (again, not this year in the case of 26 that broke a lot of things for RabbitMQ).

[noonedeadpunk]

Well, we're historically using HA queues, and switching to quorum queues is not straightforward at all. So while we're working on that upgrade, backporting to stable branches is out of question. So using latest is tricky.
We could use distro shipped packages, but we support cross- distro setup, so need to ensure same versions are available for Ubuntu/Debian/CentOS/Rocky.

But yeah, these mirrors looks quite good, and we switch to them.

[lskillen]

@michaelklishin As you say, corporate bureaucracy is a pain! At Cloudsmith we're hoping to drastically reduce our costs so that we'll be able to do something more significant for OSS in the near future. Bandwidth might be an issue right now, but I've been able to allocate you an extra 2TiB of space at Cloudsmith, so the above issue of deleting the packages is less of an issue. I hope that helps, even a little bit. :)

[michaelklishin]

@lskillen thank you, Lee. Cloudsmith and you personally have been very generous with your support of OSS RabbitMQ. Our team fully recognizes that.

All that said, we need to have more than one source of packages, and a reasonably easy way of setting up mirrors. It does not feel right that even the largest known users that have thousands of RabbitMQ clusters do not use separate mirrors.

[noonedeadpunk]

I see that PackageCloud is back now. So has smth changed wrt to it?

[noonedeadpunk]

Also, we've switched to the CloudSmith comminity repo, and it didn't took long to hit an issue with mirrors for Ubuntu:

E:Failed to fetch https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu/dists/jammy/main/binary-amd64/Packages.gz  File has unexpected size (8854 != 9044). Mirror sync in progress?
[IP: 2606:4700:130:436c:6f75:6466:6c61:7265 443]
Hashes of expected file:
 - Filesize:9044 [weak]
 - SHA256:af1cb08287c00a8866d61c432fe3c599be6d673a4a6d8b11703c7b5365f7e133
 - SHA1:4a75dc5b10e916476239481447ec4f3484b2776a [weak]
 - MD5Sum:1f241b3a3422ab495907e970e90b11a2 [weak]
 - Release file created at: Mon, 05 Jun 2023 15:37:47 +0000

That is quite annoying thing to catch in CI...

[michaelklishin]

This is due to Cloudflare caching which has since been set to 2 minutes.

Anyone is welcome to run their own mirrors and update them as frequently as they want, or use container images for CI.

[michaelklishin]

PackageCloud is again listed as an alternative on rabbitmq.com. It has a fixed monthly traffic quota
which can be exhausted. We still highly recommend using the Cloudsmith mirror where the quota is so high it is unlikely that we will hit it in the next 6-9 months.