skipping repositories when the project is known to not be there

[bewinsnw]

Our current internal repository has a problem that playing with simple-repository helped me identify - it's been very slow for a while now and I discovered that when you use its equivalent of a PrioritySelectedProjectsRepository, with our internal artifacts first and public pypi second, the 404s from the first repo take nearly a second each(!) before it moves on to check the repo where the artifact really lives. When I set up simple-repository with the same backends I saw those slow 404s, realised it wasn't a problem with SR but with the back end, but that I could hack SR to fix it.

I was able to make this all substantially faster by only looking in the repos where I know the project list/artifact could possibly exist, by checking the index page - in get_resources/get_project_page of PrioritySelectedProjectsRepository I added:

        sources = await self.get_project_sources(project_name, request_context=request_context)
        for source in sources:

and then defined that as

    async def get_project_sources(
        self,
        project_name: str,
        *,
        request_context: model.RequestContext = model.RequestContext.DEFAULT,
    ) -> dict[str, SimpleRepository]:
        now = datetime.now()
        if not (self.project_sources_expires and self.project_sources_expires > now):
            project_lists: list[model.ProjectList] = await asyncio.gather(
                *(
                    source.get_project_list(request_context=request_context)
                    for source in self.sources
                ),
                return_exceptions=True,
            )
            for project_list in project_lists:
                if isinstance(project_list, Exception):
                    # TODO: Use an exception group to raise
                    # multiple exceptions together.
                    raise project_list
            result = {}
            for source, project_list in zip(self.sources, project_lists):
                for project_detail in project_list.projects:
                    s = result.setdefault(project_detail.normalized_name, [])
                    s.append(source)
            self.project_sources = result
            self.project_sources_expires = now + timedelta(minutes = 2)
        return self.project_sources.setdefault(canonicalize_name(project_name), [])

while the bad repo I'm working around is unusually slow, this optimisation may be useful generally?

My first attempt at doing this, I began by hacking the ttl cache to accept override ttls on set/update, and made the CachedHttpRepository cache for a short time even if there was no etag (the bad repo also doesn't send etags).
But, this ran into another couple of issues which made it slower. The first problem is that I was now fetching
the index page of pypi a lot, sending a revalidation request every time (because it does send etags) and parsing
the json of the cached response every time. The 304 from pypi is quick, under 0.1s, but the parse is 0.7s.

Generally we want to cache as close as possible to the final response to avoid reprocessing; the caches in simple-repository cache the raw data instead so always need reprocessing. I think that's a mistake, but it was too big a
refactor for me today.

The other issue is when to revalidate. pip is unusual (and IMO, wrong) in ignoring cache-control expiry pypa/pip#5670 - they made a choice to reduce confusion on upload by increasing round trips for downloaders forever. uv for example doesn't do this. simple-repository revalidates every time when I think it should have a revalidate-after and expires-after ttl, which both refresh when revalidated.

To put some meat on this, the pip install --force-reinstall of 18 packages I used to test this was 28s with a cold cache; 21s when CachedHttpRepository warmed up. uv was 18s cold, and 2.2s warm. With the change above to skip the slow repo for projects that aren't in it, pip cold was 13.5s, warm was 7.7s; uv cold was 5.5s, uv warm was 1.4s (about 0.5s spent getting 304s for metadata on 3 of the packages)

[bewinsnw]

Refactored that into a RepositoryContainer so as not to pile hack upon hack into PrioritySelected:

from datetime import datetime, timedelta
from .. import errors, model
from .core import SimpleRepository, RepositoryContainer


class SkippingRepository(RepositoryContainer):
    """Proxy of a simple repository that skips projects not in its index"""

    def __init__(self,
                 source: SimpleRepository,
                 ttl_seconds: int = 60):
        self.source = source
        self.ttl_seconds = ttl_seconds
        self.project_list_expires = None
        self.project_list = None

    async def get_project_list(
        self,
        *,
        request_context: model.RequestContext = model.RequestContext.DEFAULT,
    ) -> model.ProjectList:
        now = datetime.now()
        if not (self.project_list_expires and self.project_list_expires > now):
            self.project_list = await self.source.get_project_list(request_context=request_context)
            self.project_list_expires = now + timedelta(seconds = self.ttl_seconds)
        return self.project_list

    async def get_project_page(
        self,
        project_name: str,
        *,
        request_context: model.RequestContext = model.RequestContext.DEFAULT,
    ) -> model.ProjectDetail:
        project_list = await self.get_project_list(request_context=request_context)
        if project_name in project_list:
            return await self.source.get_project_page(project_name, request_context=request_context)
        raise errors.PackageNotFoundError(
            package_name=project_name,
        )

    async def get_resource(
        self,
        project_name: str,
        resource_name: str,
        *,
        request_context: model.RequestContext = model.RequestContext.DEFAULT,
    ) -> model.Resource:
        project_list = await self.get_project_list(request_context=request_context)
        if project_name in project_list:
            return await self.source.get_resource(
                project_name,
                resource_name,
                request_context=request_context,
            )
        raise errors.ResourceUnavailable(resource_name)

The "if project_name in project_list" works because of this implementation of contains, hash, eq:

@dataclass(frozen=True)
class ProjectListElement:
    name: str  # not necessarily normalized.

    @property
    def normalized_name(self) -> str:
        return packaging.utils.canonicalize_name(self.name)

    def __hash__(self) -> int:
        return hash(self.normalized_name)

    def __eq__(self, value: object) -> bool:
        if isinstance(value, self.__class__):
            return self.normalized_name == value.normalized_name
        else:
            return False


@dataclass(frozen=True)
class ProjectList:
    """Model of the project list as described in PEP-691"""
    meta: Meta
    projects: frozenset[ProjectListElement]

    def __contains__(self, project_name: object) -> bool:
        if isinstance(project_name, str):
            return ProjectListElement(project_name) in self.projects
        return False

[pelson]

First of all, sorry for taking so long to get back to you. I saw your questions at the time, and knew they needed some in-depth thought, but have been pinned back for some time.

My first feedback is - Wow! The level to which you are digging into the implementation and finding improvements is impressive, and for me personally pretty exciting 👍.

On model improvements:

• ProjectListElement should already have __eq__ and __hash__ (as the dataclass is frozen, at

  simple-repository/simple_repository/model.py

  Line 147 in e7c06ff

  @dataclass(frozen=True)

  )
• Adding __contains__ to ProjectList is a good idea 👍. It is always possible to access project in project_list.projects, but it looks like a worthwhile API simplification 👍

There was an issue with PyPI a few months ago which resulted in very slow response times, and we also saw quite a big disruption to our service - I'm guessing that this is a similar issue to the one you describe. We are motivated to fix this, but honestly haven't been able to get on-to it recently.

From a high-level approach perspective, I would be cautious of universally using project_list to determine the existence of a project - traditionally, PyPI has been very slow to respond on project_list (I've seen multiple 10s of seconds when their CDN cache is cold), and adding such an overhead would only be desirable in certain situations. Your approach of using a repository component/subclass is the way I would go - ideally, I would also split the TTL/cache part from the containment checking part - I don't think there is a need for these two things to be coupled.

Concretely, I would advocate for a TTLCache repository, which is a pure TTL implementation (as you have above), which complements the one we already have based on TTL+HTTP invalidation. Then implementation of a ProjectExistenceBasedOnProjectList or equivalent component is fairly straight-forward. Are these something you would be interesting to have a go at implementing on the repo? We haven't got our house in order wrt. contributions, but this is something we can address if you are motivated.

[bazzargh]

in practice my caches are separated from this code - I didn't find the caching http repository flexible/reliable enough so I have this wrapped with an nginx cache on both inbound and outbound requests. The outbound requests on this endpoint cache for 10 minutes but also serve stale responses on any error or while refetching, so there is zero delay if pypi is slow; nginx is also set up to lock the cache so there's only one in-flight request for this endpoint at any one time - I don't want to DOS pypi asking for the index repeatedly, they should get one request every ten minutes at most.

Given all that, I could probably get away without the task preventing a flood of requests, but I've not seen the need yet; it'd be bad for pypi if someone deployed code like this that depends so heavily on the slow project list endpoint without having some flood protection so there's an argument for keeping those lines in.

In any case, right now the main use of this code for me is to prove to our vendor that there's a bug in their equivalent of "PrioritySelectedProjectsRepository" and having the single class makes it easy to show them.

[bewinsnw]

BTW "ProjectListElement should already have eq and hash" - yes, but it has the wrong equals and hash. It's defined on "name: str # not necessarily normalized", but I need to check the normalized name. The list could contain "bad_package" and would then claim that that repo didn't serve up "bad-package", when it does. Another choice would have been to normalize ProjectListElements on creation, but I assumed that hadn't been done for a reason.