Unable to mirror ppa1.novemberain.com

[anurag-bug]

Describe the bug

Trying to mirror rmq cloudsmith repo but it fails with forbidden error.

Reproduction steps

debmirror /var/mirror/rabbitmq-server \
          --host=ppa1.rabbitmq.com \
          --root=/rabbitmq/rabbitmq-server/deb/debian \
          --dist=bullseye,bookworm \
          --section=main \
          --arch=amd64 \
          --method=https \
          --passive \
          --diff=none \
          --ignore-release-gpg \
          --rsync-extra=none \
          --verbose \
          --nosource

Expected behavior

Should be able to mirror the repo

Additional context

Mirroring to /var/mirror/rabbitmq-server from https://[email protected]/rabbitmq/rabbitmq-server/deb/debian/
Arches: amd64
Dists: bullseye,bookworm
Sections: main
Pdiff mode: none
Passive mode on.
Will clean up after mirroring.
Attempting to get lock ...
Getting meta files ...
[ 0%] Getting: dists/bullseye/Release... Download of dists/bullseye/Release failed: 403 Forbidden
[ 0%] Getting: dists/bookworm/Release... Download of dists/bookworm/Release failed: 403 Forbidden
Errors:
Download of dists/bullseye/Release failed: 403 Forbidden
Download of dists/bookworm/Release failed: 403 Forbidden
Failed to download some Release, Release.gpg or InRelease files!

[michaelklishin]

ppa1.novemberain.com is not the Cloudsmith repo, it is our old mirror of our Cloudsmith repo.
Which was replaced in August 2024 with ppa1.rabbitmq.com.

I don't see why it would not be mirror-able with apt-mirror, there are no restrictions in place. It is also not a priority for us to make it mirror-able.

[michaelklishin]

/etc/apt/mirror.list looked something like this (it was using ppa1 but that's irrelevant, they are identical except that ppa2 is an EU-based host while ppa1 is North America-based):

############# config ##################
#
set base_path    /mnt/mirrors/apt-mirror
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  <running host architecture>
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     100
set _tilde 0
#
############# end config ##############

#
# Erlang
#

deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu lunar main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu focal main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu bionic main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu noble main

deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian buster main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bullseye main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bookworm main

deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu lunar main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu focal main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu bionic main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu noble main

deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian buster main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bullseye main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian bookworm main

#
# RabbitMQ
#

deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu lunar main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu focal main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu bionic main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu noble main

deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian buster main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bullseye main
deb-amd64 https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bookworm main

deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu lunar main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu focal main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu bionic main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu noble main

deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian buster main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bullseye main
deb-all https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian bookworm main

clean https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/ubuntu
clean https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-erlang/deb/debian
clean https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/ubuntu
clean https://ppa2.rabbitmq.com/rabbitmq/rabbitmq-server/deb/debian

The base_path is the only setting you almost certainly will try to tweak, or mount a large
enough volume (I'd recommend at least 50 GiB) at that path.

apt-mirror does not take a lot of option, we run it without any CLI arguments.

Please take it from here.

[michaelklishin]

https://www.rabbitmq.com/docs/install-debian#cloudsmith-signing-keys explains where to download the signing keys. They are also distributed via a GitHub release.

[michaelklishin]

In fact, I now recall that for a period of time, ppa2.rabbitmq.com was mirrored from ppa1.rabbitmq.com until we figured we might as well mirror once a month from the upstream when our traffic quota there rolls over.

So yeah, the same way we mirror our Cloudsmith repo — and we WOULD NOT appreciate you doing that because it would cut into the traffic quota that's already exhausted every month, which is why we have mirrors to begin with — worked for ppa1 => ppa2.

The tool we use is apt-mirror with virtually all defaults.

[michaelklishin]

I have forgotten the key limitation of our mirrors: they only offer the x86-64 builds of Erlang, no aarch64 ones (for that, there is Launchpad).

32-bit versions are not and will not be provided. aarch64 might be one day, depending on how successful our adoption of GitHub's aarch64 Linux runners will turn out.