Add available timezones

[pganssle]

Add a function to retrieve all time zones available on the system.

[FFY00]

I think maybe we should be ignoring posix and right subdirectories? Right now zoneinfo.available_timezones() returns 3 entries for each timezone, eg. Africa/Abidjan, posix/Africa/Abidjan and right/Africa/Abidjan. But only Africa/Abidjan should be used.

See https://www-master.ufr-info-p6.jussieu.fr/2014/spip.php?page=zman&nom=hwclock#sect15.

To configure a system to use a particular database all of the files located in its
directory must be copied to the root of /usr/share/zoneinfo. Files are never used directly
from the posix or ’right’ subdirectories, e.g., TZ=’right/Europe/Dublin’. This habit was
becoming so common that the upstream zoneinfo project restructured the system’s file tree
by moving the posix and ’right’ subdirectories out of the zoneinfo directory and into
sibling directories:

• /usr/share/zoneinfo
• /usr/share/zoneinfo-posix
• /usr/share/zoneinfo-leaps

zoneinfo.available_timezones() is also returning posixrules as a valid timezone, valid_key() is not enough in this case.

[FFY00]

Just a very small nitpick, this could be simplified as:

if key and valid_key(fpath):
    valid_zones.update([key])

[pganssle]

This is not quite the same thing, because valid_key actually opens and reads the file, but it won't actually change the outcome if we've already found the key from another source. If tzdata is installed, the common case is that we read a single file, populate valid_zones with all the standard keys, then start searching TZPATH to populate with any additional zones available to the user, so hopefully very few file accesses are actually happening.

[FFY00]

Ah right, I missed that, sorry.

[pganssle]

I think maybe we should be ignoring posix and right subdirectories? Right now zoneinfo.available_timezones() returns 3 entries for each timezone, eg. Africa/Abidjan, posix/Africa/Abidjan and right/Africa/Abidjan. But only Africa/Abidjan should be used.

Hm.. I am mildly torn on this, because users do have the option of choosing from the posix/ or right/ directories, and if you're using available_timezones() for something like generating hypothesis tests, you want to hit all of them.

On the other hand, the link you posted is suggestive that these are special directories and should be ignored, and make zonenames does not include the posix/ or right/ directories.

zoneinfo.available_timezones() is also returning posixrules as a valid timezone, valid_key() is not enough in this case.

This is another one I'm a bit torn about, because this actually is a valid zone, and it might be useful to people, since it seems like it's just a link to your local time zone:

>>> import zoneinfo
>>> zoneinfo.ZoneInfo("posixrules")
zoneinfo.ZoneInfo(key='posixrules')

But there should probably be a better way to get that.

I'm not sure how to solve this without hard-coding. posixrules we could detect if we were to deduplicate links, but on my system (and in the output of tzcode's make install) the links are hard links rather than symlinks to a canonical value. We'd be at risk of removing America/New_York in favor of US/Eastern (or even worse, removing both of them in favor of posixrules for people with systems set to Eastern time).

One possible option is to parse tzdata.zi and try and pull the values from there, but not every distro ships tzdata.zi, so we'd need some fallback.

The situation with posix/ and right/ are also complicated by the fact that according to your link, apparently some distros change the structure of the zoneinfo directories, so I'm not sure we can count on hard-coding posix/ and right/ as values to filter out.

If you have any suggestions let me know. I think I will send an e-mail to the tz mailing list to see if they have any ideas.

[FFY00]

I am also torn by this, it shouldn't be used, but people are using it. I think simply disallowing it might not be the best idea, but I don't think the current behavior is the best either.

From the top of my head, I would suggest adding a strict argument defaulting to True.

def available_timezones(strict=True):
	...

When strict is true we would filter out posix/ and right/. We could actually do posixrules too.

There are several use cases where you might want to use available_timezones(), and this option could be toggled accordingly. In most cases, for eg. when populating a GUI element with the available timezones, you wouldn't want posix/ and right/ to show up. But there are some use cases where we need to be more permissive, eg. when you actually need the list of all possible values, like the hypothesis tests example. Here, we can still switch on the behavior.

strict was the first thing that came to mind, it could be replaced by something better.

I'm not sure how to solve this without hard-coding. posixrules we could detect if we were to deduplicate links, but on my system (and in the output of tzcode's make install) the links are hard links rather than symlinks to a canonical value. We'd be at risk of removing America/New_York in favor of US/Eastern (or even worse, removing both of them in favor of posixrules for people with systems set to Eastern time).

I think hardcoding should be fine, at least in the context I described above.

The situation with posix/ and right/ are also complicated by the fact that according to your link, apparently some distros change the structure of the zoneinfo directories, so I'm not sure we can count on hard-coding posix/ and right/ as values to filter out.

Well, I don't think this is an issue since we only have to ignore posix/ and right/ inside TZPATH.

I think I will send an e-mail to the tz mailing list to see if they have any ideas.

Please CC me if you remember 😄

[pganssle]

From the top of my head, I would suggest adding a strict argument defaulting to True.

I dunno if this is very clear, it could mean any number of things, including, "Only give me canonical zones".

There are several use cases where you might want to use available_timezones(), and this option could be toggled accordingly. In most cases, for eg. when populating a GUI with the available timezones, you wouldn't want posix/ and right/ to show up.

Well, as I mention in the documentation, "populating a GUI with the available timezones" is not really a valid use case for this anyway. You'd want to filter the list via CLDR / ICU anyway to get human-readable names. If this is going to be a common use case, maybe it's better to avoid adding the feature at all.

I think hardcoding should be fine, at least in the context I described above.

This is destined for the Python standard library (this repo will turn into a backport when I'm done integrating it there), so I'm very reluctant to hard-code things, particularly if they are incidental and not any sort of interface with backwards-compatibility guarantees and cross-platform compatibility. People end up using old versions of Python for years before upgrading, and the release cycle is quite long.

Well, I don't think this is an issue since we only have to ignore posix/ and right/ inside TZPATH.

I was more concerned with the possibility that downstream distributors might rename these directories for whatever reason. Normally I'd say, "Ok, fine, they can just patch their version of Python", but that would mean that people not using the system python (which is my preferred recommendation anyway) would get some weird and different value here because of an unusual zoneinfo configuration.

It's a tough call and I'm tempted to push this feature to Python 3.10.

[FFY00]

I dunno if this is very clear, it could mean any number of things, including, "Only give me canonical zones".

Right. What about this:

def available_timezones(ingore=['posix', 'strict', 'zoneinfo']):
	...

Although I am conflicted by it. It would resolve most of our issues but I am not sure if it's a good api. The problem here is that we want to have a useful default for most people.

Well, as I mention in the documentation, "populating a GUI with the available timezones" is not really a valid use case for this anyway. You'd want to filter the list via CLDR / ICU anyway to get human-readable names. If this is going to be a common use case, maybe it's better to avoid adding the feature at all.

Hum, okay, that makes sense. Although I would like to note here that names like Europe/Lisbon are commonly used for timezones outside this context.

>>> import babel.dates
>>> babel.dates.get_timezone_name(babel.dates.get_timezone('Europe/Lisbon'))
'Western European Time'

If I am prompted the CLDR names, as above, personally, I would have to google my city's timezone to find out the correct name.

So, even though I think it would be great to be able to just use CLDR, I don't think it would be very realistic to expect it from everyone. If I was designing a UI I would probably go for Europe/Lisbon (Western European Time).

I think this is still a valid use-case.

This is destined for the Python standard library (this repo will turn into a backport when I'm done integrating it there), so I'm very reluctant to hard-code things, particularly if they are incidental and not any sort of interface with backwards-compatibility guarantees and cross-platform compatibility. People end up using old versions of Python for years before upgrading, and the release cycle is quite long.

Yeah, I am aware, I got here from bpo. Worst case scenario people could opt out and do their own filtering, but yes, it would be great to avoid any issues.

I was more concerned with the possibility that downstream distributors might rename these directories for whatever reason. Normally I'd say, "Ok, fine, they can just patch their version of Python", but that would mean that people not using the system python (which is my preferred recommendation anyway) would get some weird and different value here because of an unusual zoneinfo configuration.

I don't think anyone is doing this, they may be changing the paths, but renaming the posix and strict folder inside TZPATH is a different thing. Anyway, I have no backing for this.