[feature] Automatic migration for config files

[hikari-no-yume]

Is your feature request related to a problem ?

I want to keep my server software up to date, but editing my config file with major version updates is a bit of a chore, and the main thing holding me back from doing it more often.

Describe the solution you'd like.

It should be possible for GoToSocial to do migrations for config files in the same way it (presumably) does for the database.

A new, mandatory key can be added to config.yaml indicating which version the file was written for.

Then, when GoToSocial starts, it can check if that key's value is less than the current version, and do non-destructive editing to rename or add config keys as appropriate. Ideally, it might even add the explanatory comments from the example config file.

If it's crucial the user themselves decides what to do with a setting, maybe they can be prompted interactively somehow.

Describe alternatives you've considered.

A simpler alternative would be to provide a summary, for each release, of all the config key changes. But that would still make it a bit of a chore.

Additional context.

No response

[tsmethurst]

Lord, this sounds like a huge feature to implement and maintain, and not something I think we have time for at the moment. Writing to the admin's config file also brings with it the risk of accidentally corrupting it or overwriting it, as well.

While we're still in alpha, it's likely that the config file will change regularly as we continue to tweak stuff. I imagine once we're in beta this will stabilize somewhat, but for now it's part of the cost of using alpha software, in my view. It's annoying, yes, but we do try to mitigate this by linking to a diff of the config file with every release.

It's also worth bearing in mind that in many cases, because of the defaults that we set, it's not even necessary for server admins to actually change anything: if a key is missing in their config file because they didn't update it, GtS will just fall back to using defaults. In the case of using Docker to run GtS, you don't even need a config file unless you're tweaking advanced stuff, you can just use environment variables (which seldom get renamed). Perhaps we could better document this, come to think of it..

[hikari-no-yume]

Writing to the admin's config file also brings with it the risk of accidentally corrupting it or overwriting it, as well.

Maybe I shouldn't have said fully automatic, it seems like a bad idea both from a safety and complexity angle. I guess it should save a backup first and prompt the user before writing anything.

Lord, this sounds like a huge feature to implement and maintain, and not something I think we have time for at the moment.

Is it really so huge? The core of it could be as simple as this semi-realistic pseudo-code:

CONFIG_KEY_HISTORY = [
    # (required_in_version, key_name, key_description, key_default)
    ("0.9.3", "foo-bar", "This must be set to true if the foo will be bar'd", "false"),
    ("0.9.7", "qux-size", "Maximum size of qux, in bytes", "1000000"),
]

if CURRENT_VERSION < config["version"]:
    print("Warning: config file is for a newer version")
elif CURRENT_VERSION > config["version"]:
    if not upgrade_mode:
        exit()
    for required_in_version, key_name, key_description, key_default in CONFIG_KEY_HISTORY:
       if config["version"] < required_in_version && key_name not in config:
           print("New config option: " + key_name + " (version: " + required_in_version + ")")
           print("Description: " + key_description)
           print("Default: " + key_default)
           value = prompt("Choose a value (or press Enter for default): ")
           if value == "":
               value = key_default
           append_config_option(key_description, key_name, value)
    change_config_option("version", CURRENT_VERSION)

append_config_option() would just spit out lines at the end of the file. change_config_option() is the only complicated bit, but it would only be used to change the version number, so it could use a very simple strategy.

Maybe I'm getting ahead of myself but I'd be willing to try to implement it myself.

[mirabilos]

Oh please no, I’d rather manually review the new options and configure them (and I do that before I even take the old version offline and do a DB backup, so I have everything in place already before moving the old version away and the new version into its place).

[daenney]

From my point of view, I'd also rather we don't do this. As @tsmethurst mentioned, we're still alpha. That comes will all kinds of caveats, configuration changes and potentially bumpy upgrades amongst them. The pseudo code might not seem like a lot of work right now, but over time that accumulates. I'm also guessing it's not going to be as straightforward as defaults for existing values also change. This is something we'll need to maintain in perpetuity in order to allow people to upgrade-jump past multiple versions too.

Even for single instances, folks use tools to manage the configuration, like Ansible for example. That really doesn't play well with us automatically doing anything. At best, I'd want to implement this as it generating a diff between the current config and the new values and give an admin the ability to apply that diff manually. They can patch that directly into a new config, apply it to their Ansible repo, however they're managing their config with NixOS etc.

I think the reason this ends up looking like a lot of work in GtS right now is because we tell people to copy our defaults into their config. Which also means that whenever we tweak them, people need to copy them over. But since those are default values, that's not actually necessary unless people want to change them away from the default. In reality you're probably tweaking <10 config options and leaving the rest for what they are. Taking a look at my own instance, these are the ones I've changed:

• landing-page-user, because I have a single user instance
• host, everyone needs to change this one
• account-domain, because I run a split-domain setup
• bind-address, since I have a reverse proxy in front of it
• port and trusted-proxies, for the same reason
• db-type and db-address, because I roll with SQLite
• web-template-base-dir and web-asset-base-dir because I don't run from /gotosocial
• accounts-registration-open, because I don't want that enabled by default
• media-description-min-chars, because accessibility matters
• media-remote-cache-days, but [feature] Lower remote media cache config duration #2007 means I no longer have to
• storage-local-base-path, for the same reason as the web-* values
• statuses-media-max-files, because there's such a thing as too many images

Most people won't even need to tweak all of these. Some folks will need to tweak the letsencrypt-*, db-* and storage-* more depending on how they run their instance. In all of the cases, I've set this value once and never had to bother with it again. None of the other values matter, and I could remove them from my config entirely as they'll be initialised with default.go.

I think if we change our guidance from "copy all the values into your config" to "only put values in your config.yaml where you need something other than the default" most of this problem goes away.