Possibly broken EOL chroots removal

[FrostyX]

From matrix:

solopasha
Is removing outdated repositories broken? I clicked "Expire now" for this repo https://copr.fedorainfracloud.org/coprs/solopasha/nautilus-code/ in https://copr.fedorainfracloud.org/user/repositories/ some time ago and "Remaining time" shows -31 days

[FrostyX]

From the logs:

2025-01-20 03:27:28,873 [ERROR][/usr/share/copr/coprs_frontend/commands/delete_outdated_chroots.py:32|delete_outdated_chroots:delete_outdated_chroots_function][SERVER] Refusing to delete solopasha/nautilus-code/fedora-39-x86_64 because any notification was sent about its deletion

There are ~100 chroots there weren't deleted

Refusing to delete @meta/netconsd/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete davidsch/preview/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete davidsch/scripts/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/ebranch/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete solopasha/torrserver/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/ebranch/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete solopasha/nautilus-code/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete solopasha/torrserver/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete solopasha/nautilus-code/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/rust-coreutils/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-s390x because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-ppc64le because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-s390x because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-aarch64 because any notification was sent about its deletion
Refusing to delete salimma/rust-coreutils/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-x86_64 because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-x86_64 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-ppc64le because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-aarch64 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-i386 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/test/fedora-38-x86_64 because any notification was sent about its deletion
Refusing to delete davidsch/review/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete davidsch/review/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete davidsch/preview/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete davidsch/scripts/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/ebranch/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete solopasha/torrserver/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/ebranch/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete solopasha/nautilus-code/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete @meta/go2chef/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete @meta/drgn/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-s390x because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete solopasha/torrserver/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete solopasha/nautilus-code/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete salimma/rust-coreutils/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-s390x because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-ppc64le because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-s390x because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-aarch64 because any notification was sent about its deletion
Refusing to delete salimma/rust-coreutils/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete @meta/netconsd/fedora-39-x86_64 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete salimma/golang-wip/fedora-39-i386 because any notification was sent about its deletion
Refusing to delete @meta/pystemd/fedora-39-ppc64le because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-x86_64 because any notification was sent about its deletion
Refusing to delete poscat/mdbook-linkcheck/fedora-38-x86_64 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-ppc64le because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-aarch64 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/fusion-remi/fedora-38-i386 because any notification was sent about its deletion
Refusing to delete goddabrouquita8280/test/fedora-38-x86_64 because any notification was sent about its deletion
Refusing to delete davidsch/review/fedora-39-aarch64 because any notification was sent about its deletion
Refusing to delete davidsch/review/fedora-39-x86_64 because any notification was sent about its deletion

[FrostyX]

One observation is that when a user click the "Expire now" button, we call this function

copr/frontend/coprs_frontend/coprs/logic/outdated_chroots_logic.py

Lines 76 to 83 in 2075cac

	@classmethod
	def expire(cls, copr_chroot):
	"""
	A `user` decided to expire some EOL chroot,
	i.e. its data should be deleted ASAP
	"""
	delete_after_days = app.config["EOL_CHROOTS_EXPIRE_PERIOD"]
	cls._update_copr_chroot(copr_chroot, delete_after_days)

which sets that the data should be deleted in 12 hours. We send the notifications daily through Cron, so this doesn't overlap perfectly, and a deletion can be delayed by a day.

But that is not the problem here. I am manually running

runuser -c 'copr-frontend notify_outdated_chroots' - copr-fe

and it doesn't set the delete_notify in the database.

[FrostyX]

I think the problem is here:

copr/frontend/coprs_frontend/coprs/logic/coprs_logic.py

Lines 1270 to 1271 in fde1bd2

	return (query.filter(models.CoprChroot.delete_after
	>= datetime.datetime.now())

The notification script never tries to notify chroots that should have already been deleted. Hence ... we are deadlocked.

[FrostyX]

In the PR #3599 I am proposing a fix that will prevent this situation in the future.

Regarding the 94 projects that are currently in a weird state:

• 74 of them are F39 - which could be scheduled for deletion so soon only if the project owners clicked the "Expire now" button, because by default F39 chroots still have 135 days until they are mass deleted. I suggest updating the database and specifying the delete_notify to NOW() so that they are deleted during the next Cron run
• 20 of them are F38 - but only owned by two users, goddabrouquita8280 and poscat. I suggest contacting them directly and figuring out what can be deleted and what not. We can then set the delete_notify to the timestamp of the discussion

[FrostyX]

I hotfixed this in production, so this shouldn't happen for new chroots. And I did this database magic to resolve the F38 and F39 issues.

F39:

coprdb=# select id from mock_chroot where os_release='fedora' and os_version='39';
┌─────┐
│ id  │
├─────┤
│ 193 │
│ 192 │
│ 191 │
│ 194 │
│ 190 │
└─────┘
(5 rows)

coprdb=# select count(id) from copr_chroot where mock_chroot_id in (193, 192, 191, 194, 190) and delete_notify is null and delete_after < now();
┌───────┐
│ count │
├───────┤
│   123 │
└───────┘
(1 row)

coprdb=# update copr_chroot set delete_notify=now() where mock_chroot_id in (193, 192, 191, 194, 190) and delete_notify is null and delete_after < now();
UPDATE 123

F38:

coprdb=# select id from public.user where username='poscat';
  id
-------
 11316
(1 row)

coprdb=# select id from copr where user_id=11316 and name in ('mdbook-linkcheck');
   id
--------
 131105
(1 row)

coprdb=# select id from public.user where username='goddabrouquita8280';
  id
-------
 11298
(1 row)

coprdb=# select id from copr where user_id=11298 and name in ('fusion-remi', 'test');
   id
--------
 130879
 130934
(2 rows)

coprdb=# select id from mock_chroot where os_release='fedora' and os_version='38';
 id
-----
 181
 182
 180
 183
 184
 179
(6 rows)

coprdb=# select id from copr_chroot where copr_id in (131105, 130879, 130934) and mock_chroot_id in (181, 182, 180, 183, 184, 179) and delete_after < now();
   id
--------
 736134
 736425
 737281
 736426
 736428
 737284
 736427
 737283
 736415
 737271
(10 rows)

coprdb=# update copr_chroot set delete_after='2025-02-27 13:59:04.180209' where copr_id in (131105, 130879, 130934) and mock_chroot_id in (181, 182, 180, 183, 184, 179) and delete_after < now();
UPDATE 10

Helper for calculating the date

>>> from datetime import datetime, timedelta
>>> str(datetime.now() + timedelta(days=30))
'2025-02-27 13:59:04.180209'

I think this is finished now.