Countries.__iter__ very slow in Django admin

[christianglodt]

I see a strange performance issue with django-countries 7.5.1. I'm using Django 4.2.8, I have USE_I18N on and I do not have pyuca installed.

In my app there's a django model with a CountryField. Its admin page includes the country field in list_display and list_filter. For some reason, rendering the changelist page takes upward of 6-7 seconds (on a Ryzen 9 5900X CPU). If I remove the country field from list_display and list_filter, the same list renders in around 300ms. This is all with DEBUG on, on the regular development runserver.

I've done a bit of profiling, and found that it seems to be Countries.__iter__ which is slow. I'm not sure how to share the profiling data, so here's a screenshot of a flame graph:

Please note that according to the profiler, the top entry for Countries.__iter__ in the flamegraph above corresponds to an execution time of +- 3.4 seconds.

For verification I tried listing the countries in a Python shell (shell_plus from django-extensions) instead, and there the slowdown doesn't happen:

Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django_countries import countries
>>> import timeit
>>> timeit.timeit('list(countries)', globals=locals(), number=1)
0.004178638999746909
>>> 

So this indicates to me that there may be a problem having to do with my Django configuration, in particular wrt. translations. But I haven't managed to find out more yet.

I'd be grateful for any help with this issue.

[SmileyChris]

Timing the list of countries still wouldn't be triggering the translations of the lazy strings. You'd probably need to time something like [str(c) for c in countries]

[christianglodt]

Thanks for your reply. I've investigated a bit more, and I've found that the slowness I observe may have to do with the fact that the Countries.__iter__() method is being called many times, and not with a single call being particularly slow.

My testing in the Python interpreter did not reflect that and called it only once, and is thus not realistic.

With the CountryField in list_display, I observe that the Countries.__iter__() method is called twice as many times as the number of instances displayed in the admin changelist view. I've seen this by putting a breakpoint at the start of the __iter__ method, and I got the following stack trace (many times):

__iter__ (/opt/venv/lib/python3.11/site-packages/django_countries/__init__.py:342)
_get_flatchoices (/opt/venv/lib/python3.11/site-packages/django/db/models/fields/__init__.py:1025)
display_for_field (/opt/venv/lib/python3.11/site-packages/django/contrib/admin/utils.py:406)
items_for_result (/opt/venv/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py:235)
__init__ (/opt/venv/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py:303)
results (/opt/venv/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py:312)
result_list (/opt/venv/lib/python3.11/site-packages/django/contrib/admin/templatetags/admin_list.py:336)
render (/opt/venv/lib/python3.11/site-packages/django/template/library.py:258)
render (/opt/venv/lib/python3.11/site-packages/django/contrib/admin/templatetags/base.py:45)
render_annotated (/opt/venv/lib/python3.11/site-packages/django/template/base.py:966)
<listcomp> (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/loader_tags.py:63)
render_annotated (/opt/venv/lib/python3.11/site-packages/django/template/base.py:966)
<listcomp> (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/loader_tags.py:63)
super (/opt/venv/lib/python3.11/site-packages/django/template/loader_tags.py:79)
_resolve_lookup (/opt/venv/lib/python3.11/site-packages/django/template/base.py:914)
resolve (/opt/venv/lib/python3.11/site-packages/django/template/base.py:847)
resolve (/opt/venv/lib/python3.11/site-packages/django/template/base.py:715)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1064)
render_annotated (/opt/venv/lib/python3.11/site-packages/django/template/base.py:966)
<listcomp> (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/loader_tags.py:63)
render_annotated (/opt/venv/lib/python3.11/site-packages/django/template/base.py:966)
<listcomp> (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
instrumented_test_render (/opt/venv/lib/python3.11/site-packages/django/test/utils.py:112)
render (/opt/venv/lib/python3.11/site-packages/django/template/loader_tags.py:157)
render_annotated (/opt/venv/lib/python3.11/site-packages/django/template/base.py:966)
<listcomp> (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
instrumented_test_render (/opt/venv/lib/python3.11/site-packages/django/test/utils.py:112)
render (/opt/venv/lib/python3.11/site-packages/django/template/loader_tags.py:157)
render_annotated (/opt/venv/lib/python3.11/site-packages/django/template/base.py:966)
<listcomp> (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)
render (/opt/venv/lib/python3.11/site-packages/django/template/base.py:1005)

It seems Django calls display_for_field for every instance in the changelist, which calls _get_flatchoices, which iterates over all choices provided by Countries.__iter__(). The choices are translated and sorted every time in Countries.__iter__().

It would be great if it would be possible to cache the result.

[christianglodt]

Just as an additional data point, if I make a crude custom CountryField like this:

class CachingCountryField(CountryField):
    def __init__(self, *args, **kwargs):
        super().__init__(self, *args, **kwargs)
        self.choices = list(self.choices)

With this, the page rendering time for the changelist view in my development server goes down from +-10.000 ms to +-450ms.

Obviously this is not a real solution. But I think it shows that caching would be worthwhile.

[SmileyChris]

Thanks for your findings! I'll take another dive into why this is happening soon. Caching would be a potential way to bandaid fix this and probably a good efficiency improvement anyway, however the caching does need to take into account the current active translation and not just hard code the list (hence why it's being done lazily currently)

[ronny-rentner]

I have unfortunately the same problem. Just posted the same report in another, 10 year old issue thread as I found that earlier.

Showing a country column in admin makes it 10 times slower than without, so an admin change list view takes 700 ms CPU time instead of 70 ms according to django-debug-toolbar.

I am using django-countries version 7.6.1.

[ronny-rentner]

Ok, I think my problem is a different one. @SmileyChris

I've drilled it down to fields.py and the following code in the __init__() method:

        if django.VERSION >= (5, 0):
            # Use new lazy callable support
            kwargs["choices"] = lambda: self.countries
        else:
            kwargs["choices"] = self.countries

The use of that lambda function is causing the slowdown for me. If I comment that block, it's fast and the problem disappears.

[eillarra]

Ok, I think my problem is a different one. @SmileyChris

I've drilled it down to fields.py and the following code in the __init__() method:

        if django.VERSION >= (5, 0):
            # Use new lazy callable support
            kwargs["choices"] = lambda: self.countries
        else:
            kwargs["choices"] = self.countries

The use of that lambda function is causing the slowdown for me. If I comment that block, it's fast and the problem disappears.

I think this is not only a speed issue: in my case, development always works (even if you can see that showing the country field in the list is somewhat slower than not showing it) but on production the django ~5.0 + django-countries ~7.6 combo just System Exit's (using gunicorn+gevent) -- when loading the changelist page in the admin area.

I am guessing that the use of lambda here has some extra implications, maybe because asgiref is used at some point for lazy loading and that kills gunicorn (this is just a guess, below you can find the stack trace).

Stack Trace (`django ~5.0` + `django-countries ~7.6` + `gunicorn+gevent`)

SystemExit: 1
  File "django/core/handlers/wsgi.py", line 124, in __call__
    response = self.get_response(request)
  File "django/core/handlers/base.py", line 140, in get_response
    response = self._middleware_chain(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "corsheaders/middleware.py", line 56, in __call__
    result = self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "whitenoise/middleware.py", line 124, in __call__
    return self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "allauth/account/middleware.py", line 29, in middleware
    response = get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 129, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 220, in _get_response
    response = response.render()
  File "django/template/response.py", line 114, in render
    self.content = self.rendered_content
  File "django/template/response.py", line 92, in rendered_content
    return template.render(context, self._request)
  File "django/template/backends/django.py", line 107, in render
    return self.template.render(context)
  File "django/template/base.py", line 171, in render
    return self._render(context)
  File "django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "django/template/base.py", line 1008, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "django/template/base.py", line 969, in render_annotated
    return self.render(context)
  File "django/template/loader_tags.py", line 159, in render
    return compiled_parent._render(context)
  File "django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "django/template/base.py", line 1008, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "django/template/base.py", line 969, in render_annotated
    return self.render(context)
  File "django/template/loader_tags.py", line 159, in render
    return compiled_parent._render(context)
  File "django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "django/template/base.py", line 1008, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "django/template/base.py", line 969, in render_annotated
    return self.render(context)
  File "django/template/loader_tags.py", line 65, in render
    result = block.nodelist.render(context)
  File "django/template/base.py", line 1008, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "django/template/base.py", line 969, in render_annotated
    return self.render(context)
  File "django/template/loader_tags.py", line 65, in render
    result = block.nodelist.render(context)
  File "django/template/base.py", line 1008, in render
    return SafeString("".join([node.render_annotated(context) for node in self]))
  File "django/template/base.py", line 969, in render_annotated
    return self.render(context)
  File "django/contrib/admin/templatetags/base.py", line 45, in render
    return super().render(context)
  File "django/template/library.py", line 258, in render
    _dict = self.func(*resolved_args, **resolved_kwargs)
  File "django/contrib/admin/templatetags/admin_list.py", line 346, in result_list
    "results": list(results(cl)),
  File "django/contrib/admin/templatetags/admin_list.py", line 322, in results
    yield ResultList(None, items_for_result(cl, res, None))
  File "django/contrib/admin/templatetags/admin_list.py", line 313, in __init__
    super().__init__(*items)
  File "django/contrib/admin/templatetags/admin_list.py", line 243, in items_for_result
    result_repr = display_for_field(value, f, empty_value_display)
  File "django/contrib/admin/utils.py", line 432, in display_for_field
    if getattr(field, "flatchoices", None):
  File "django/db/models/fields/__init__.py", line 1090, in flatchoices
    return list(flatten_choices(self.choices))
  File "django/utils/choices.py", line 64, in flatten_choices
    for value_or_group, label_or_nested in choices or ():
  File "django/utils/choices.py", line 59, in __iter__
    yield from normalize_choices(self.func())
  File "django/utils/choices.py", line 109, in normalize_choices
    return [(k, normalize_choices(v, depth=depth + 1)) for k, v in value]
  File "__init__.py", line 359, in __iter__
    countries = tuple(
  File "__init__.py", line 287, in translate_code
    yield self.translate_pair(code, name)
  File "__init__.py", line 321, in translate_pair
    country_name = force_str(name)
  File "django/utils/encoding.py", line 69, in force_str
    s = str(s)
  File "django/utils/functional.py", line 119, in __str__
    return str(self.__cast())
  File "django/utils/functional.py", line 110, in __cast
    return func(*self._args, **self._kw)
  File "django/utils/translation/__init__.py", line 96, in gettext
    return _trans.gettext(message)
  File "django/utils/translation/trans_real.py", line 384, in gettext
    result = translation_object.gettext(eol_message)
  File "gettext.py", line 437, in gettext
    return self._fallback.gettext(message)
  File "__init__.py", line 61, in gettext
    if not getattr(_translation_state, "fallback", True):
  File "asgiref/local.py", line 117, in __getattr__
    with self._lock_storage() as storage:
  File "contextlib.py", line 299, in helper
    @wraps(func)
  File "gunicorn/workers/base.py", line 204, in handle_abort
    sys.exit(1)

[ronny-rentner]

Have you tried removing the lambda?

[eillarra]

Yes, the latest django-countries code seems to work if you only remove the lambda:
eillarra@02f6da7 (this is a fast fork of the main branch)

[yanivtoledano]

I'm seeing the same thing FWIW - adding a country field to an admin list view in list_display increases the page load time >10x for me (from 80ms to 1.2s).