Add script to clean up translations

[balloob]

Breaking change

Proposed change

Over time we change our strings.json file and we often forget to update Lokalise to either rename the keys or delete them. This results in us having stale keys in Lokalise, which we then download and include in our releases. These translations are then sent to the frontend too, where they sit in memory.

This script will find all keys that are in the download from Lokalise, which are no longer represented in the base strings file. These keys are removed.

Here is an output when I just ran it. Note that the first key can no longer be found in Lokalise as I had used that key to test.

We could consider including this in our nightly translations bump, but I rather not, because sometimes we do plan on renaming keys.

@bramkragten this might be useful for the frontend too.

› python3 script/translations_clean.py
Processing component::deconz::config::step::hassio_confirm::data::allow_clip_sensor
Lookin up key in Lokalise returns 0 results, expected 1
Processing component::deconz::config::step::hassio_confirm::data::allow_deconz_groups
Processing component::deconz::config::step::options::data::allow_clip_sensor
Processing component::deconz::config::step::options::data::allow_deconz_groups
Processing component::deconz::config::step::options::title
Processing component::deconz::options::step::async_step_deconz_devices::data::allow_clip_sensor
Processing component::deconz::options::step::async_step_deconz_devices::data::allow_deconz_groups
Processing component::deconz::options::step::async_step_deconz_devices::description
Processing component::ambient_station::config::error::identifier_exists
Processing component::geonetnz_quakes::config::error::identifier_exists
Processing component::vizio::config::abort::already_in_progress
Processing component::vizio::config::abort::already_setup_with_diff_host_and_name
Processing component::vizio::config::abort::host_exists
Processing component::vizio::config::abort::name_exists
Processing component::vizio::config::abort::updated_options
Processing component::vizio::config::abort::updated_volume_step
Processing component::vizio::config::error::tv_needs_token
Processing component::vizio::config::step::tv_apps::data::apps_to_include_or_exclude
Processing component::vizio::config::step::tv_apps::data::include_or_exclude
Processing component::vizio::config::step::tv_apps::description
Processing component::vizio::config::step::tv_apps::title
Processing component::vizio::config::step::user_tv::data::apps_to_include_or_exclude
Processing component::vizio::config::step::user_tv::data::include_or_exclude
Processing component::vizio::config::step::user_tv::description
Processing component::vizio::config::step::user_tv::title
Processing component::vizio::options::step::init::data::timeout
Processing component::cert_expiry::config::abort::host_port_exists
Processing component::cert_expiry::config::error::certificate_error
Processing component::cert_expiry::config::error::certificate_fetch_failed
Processing component::cert_expiry::config::error::host_port_exists
Processing component::cert_expiry::config::error::wrong_host
Processing component::simplisafe::config::step::user::data::code
Processing component::withings::config::abort::no_flows
Processing component::withings::config::step::user::data::profile
Processing component::withings::config::step::user::description
Processing component::withings::config::step::user::title
Processing component::harmony::config::error::invalid_auth
Processing component::wwlln::config::abort::window_too_small
Processing component::wwlln::config::error::identifier_exists
Processing component::samsungtv::config::abort::not_found
Processing component::binary_sensor::device_automation::trigger_type::closed
Processing component::binary_sensor::device_automation::trigger_type::moist§
Processing component::notion::config::error::identifier_exists
Processing component::airvisual::config::step::user::data::show_on_map
Processing component::opentherm_gw::config::step::init::data::floor_temperature
Processing component::opentherm_gw::config::step::init::data::precision
Processing component::axis::config::abort::updated_configuration
Processing component::minecraft_server::config::step::user::data::port
Processing component::switch::device_automation::condition_type::turn_off
Processing component::switch::device_automation::condition_type::turn_on
Processing component::airly::config::error::name_exists
Processing component::plex::config::abort::discovery_no_file
Processing component::plex::config::error::no_token
Processing component::plex::config::step::manual_setup::data::host
Processing component::plex::config::step::manual_setup::data::port
Processing component::plex::config::step::manual_setup::data::ssl
Processing component::plex::config::step::manual_setup::data::token
Processing component::plex::config::step::manual_setup::data::verify_ssl
Processing component::plex::config::step::manual_setup::title
Processing component::plex::config::step::user::data::manual_setup
Processing component::plex::config::step::user::data::token
Processing component::plex::config::step::user::description
Processing component::plex::config::step::user::title
Processing component::plex::options::step::plex_mp_settings::data::show_all_controls
Processing component::transmission::config::abort::one_instance_allowed
Processing component::transmission::config::step::options::data::scan_interval
Processing component::transmission::config::step::options::title
Processing component::transmission::options::step::init::description
Processing component::roku::config::error::unknown
Processing component::directv::config::error::unknown
Deleting keys: 20415195, 10125943, 10525352, 10125944, 27063585, 27063586, 27063587, 17440810, 27063588, 35122573, 35304772, 35122575, 35122576, 35287929, 35122577, 35122581, 38503717, 38503718, 38503719, 38503720, 38397197, 38397198, 38397199, 38397200, 35287930, 27063575, 29555589, 27063576, 27063578, 29555590, 13713563, 27862136, 27147398, 27147399, 27147400, 39613299, 38331420, 24627498, 34683255, 28043843, 28043850, 24449374, 38073804, 28601769, 28601772, 34386413, 36567775, 27539919, 27539920, 28541840, 28777946, 28044847, 28044848, 28044849, 28044850, 28044851, 28044852, 28044853, 28044854, 27946923, 27946924, 27946925, 28210592, 28210639, 28210642, 28210643, 28210652, 39197497, 38961181
{'project_id': '130246255a974bd3b5e8a1.51616605', 'keys_removed': True, 'keys_locked': 0}

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

[MartinHjelmare]

Maybe have find_extra return the missing_keys to avoid side effects? It does add some complexity due to calling itself.

[balloob]

It means that we keep copying lists around. I prefer to keep it like this. Generally for recursion one would have 1 "main" function that creates a list and a second recursion function that just operates on that list. Here I skipped the main function.