failing repositories getting disabled

[Aleksey-Maksimov]

Hello

Sometimes some repositories are not updated.

# /usr/bin/patchman --refresh-repos --force --repo 137

Refreshing metadata for Repo 137
Repository 137 : Docker - Debian 12 bookworm stable binary-amd64
Found deb repo - https://download.docker.com/linux/debian/dists/bookworm/stable/binary-amd64/Packages.bz2
Downloading repo info:             100%|##############################################################################################################################################################################|Time: 0:00:00
/usr/lib/python3/dist-packages/debian/deb822.py:734: UserWarning: Parsing of Deb822 data with python3-apt's apt_pkg was requested but this cannot be done on non-file input.                                         |ETA:  --:--:--
  warnings.warn(msg)
Extracting packages:               100%|##############################################################################################################################################################################|Time: 0:00:00
Fetching existing packages:        100%|##############################################################################################################################################################################|Time: 0:00:00

Please tell me how to diagnose the cause of the problem.

[furlongm]

That is just a warning message from python3-debian, it is not an error.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913274 for the background to the warning.

[Aleksey-Maksimov]

This link says that the problem has long been fixed in the python3-debian 0.1.34 version. Now I have the current version 0.1.49 installed on my Debian 12 system.
Besides, this does not solve the problem that some of our repositories are not updated periodically. How can we understand why they are not updated?

[furlongm]

It is just a warning, it is not related to your repos failing. Do you have the log output where the repos are failing?

[Aleksey-Maksimov]

The problem with updating some repositories is floating. One day the same repository can update successfully, and on another day it does not update. For example, today at 7 am using the systemd timer, an update of the Patchman server was launched with the command "/usr/bin/patchman --all --force"

Now the web interface of the Patchman server shows that 2 repositories have mirrors that are not responding

However, the log does not show any obvious update errors for these repositories.

...
Dec 25 07:15:07 PM-SRV patchman[387757]: Repository 109 : Linux Debian 12 bookworm-backports contrib binary-amd64
Dec 25 07:15:07 PM-SRV patchman[387757]: Found deb repo - http://deb.debian.org/debian/dists/bookworm-backports/contrib/binary-amd64/Packages.xz
Dec 25 07:15:07 PM-SRV patchman[387757]: [212B blob data]
Dec 25 07:15:07 PM-SRV patchman[387757]: [2.2K blob data]
Dec 25 07:15:07 PM-SRV patchman[387757]: [2.2K blob data]
Dec 25 07:15:07 PM-SRV patchman[387757]: Repository 110 : Linux Debian 12 bookworm-backports main binary-amd64
Dec 25 07:15:07 PM-SRV patchman[387757]: Found deb repo - http://deb.debian.org/debian/dists/bookworm-backports/main/binary-amd64/Packages.xz
Dec 25 07:15:07 PM-SRV patchman[387757]: [1.3K blob data]
Dec 25 07:15:08 PM-SRV patchman[387757]: [6.7K blob data]
Dec 25 07:15:08 PM-SRV patchman[387757]: [6.7K blob data]
Dec 25 07:15:08 PM-SRV patchman[387757]: [283B blob data]
Dec 25 07:15:08 PM-SRV patchman[387757]: [283B blob data]
...

[Aleksey-Maksimov]

Is it possible to somehow enable debugging of repository update processing?

[furlongm]

Are there no progressbars running for the OS updates?

[Aleksey-Maksimov]

There is a progress bar if we run the update command manually:

# /usr/bin/patchman --refresh-repos --force --repo 78
Refreshing metadata for Repo 78
Repository 78 : Patchman - Debian 12 bookworm main binary-amd64
Found deb repo - https://repo.openbytes.ie/patchman/debian/dists/bookworm/main/binary-amd64/Packages.xz
Downloading repo info:             100%|###################################################################################################################################################################|Time: 0:00:00
/usr/lib/python3/dist-packages/debian/deb822.py:734: UserWarning: Parsing of Deb822 data with python3-apt's apt_pkg was requested but this cannot be done on non-file input.                              |ETA:  --:--:--
  warnings.warn(msg)
Extracting packages:               100%|###################################################################################################################################################################|Time: 0:00:00
Fetching existing packages:        100%|###################################################################################################################################################################|Time: 0:00:00

But as we can see, the command is executed instantly, and the repository data is not actually updated, even though we specified the --force option.

[Aleksey-Maksimov]

The bad thing about this whole situation is that the mirror properties have a counter of unsuccessful updates, which when it reaches 28 simply makes the mirror useless because it stops updating it. This is a very strange decision, since nothing is written about this in the patchman documentation and this limit is not put into the configuration file, but simply hardcoded.

And as we understand, even with a successful update, this counter of unsuccessful updates is not reset. If we have an unstable Internet and the repository update process cannot update the mirrors from time to time, then over time (when the counter reaches 28) this leads to the fact that all mirrors will quietly stop updating at all. Correct me if I'm wrong.

[furlongm]

You are correct. IIRC the original logic was something like "if an upstream mirror is unavailable for a month, it should be disabled or replaced with a better mirror, and this will help people notice". Happy to add a configurable value here instead and/or to reset the fail count if a mirror if it becomes available again.

[furlongm]

Could also represent the fail count as a percentage of access failures, and have the mirror disabled upon reaching a certain failure threshold?

[Aleksey-Maksimov]

I don't see any point in disabling the mirror. If our Internet provider or Internet access equipment is unstable, the mirror may be unavailable periodically. In this case, the counter will increase, as is happening with us now. The administrator already sees in the dashboard that there are repositories with which there are accessibility problems. In my opinion, this is enough.

[furlongm]

The original use-case was for folks using mirrorlists where mirrors periodically drop off completely and don't come back. In that case, it makes sense to disable the mirror and pick another mirror from the mirrorlist. This requires no intervention or monitoring from the user.

Your use-case is one not considered before, but it also makes sense. I have added a configurable in #635 if you would like to test it?

[Aleksey-Maksimov]

Your use-case is one not considered before, but it also makes sense. I have added a configurable in #635 if you would like to test it?

Can you please build a new version of the deb package with these changes?