♻️ director-v2 created api-keys are removed if not used by any service ⚠️

[GitHK]

What do these changes do?

When director-v2 creates a service which requires an API-key, director-v2 will also create the API-key.
If the service is forcefully removed (eg: manually from portioner) or in case of errors, the API-Key remain in the system.

This was fixed by keeping track of the created API-keys and adding a backup background task that checks if any keys require cleanup.

• ♻️ the DistributedIdentifier is keeps track of created identifiers in Redis
• ✨ ⚠️ added new Redis database, number 6
• reformatted the redis-commander's REDIS_HOSTS env var. It is now multiline and redis commander stil likes it even though in the end it gets translated to REDIS_HOSTS= resources:redis:6379:0, locks:redis:6379:1, validation_codes:redis:6379:2, scheduled_maintenance:redis:6379:3, user_notifications:redis:6379:4, announcements:redis:6379:5, distributed_identifiers:redis:6379:6 form yaml

Related issue/s

How to test

Dev Checklist

• No ENV changes or I properly updated ENV (read the instruction)
• ⚠️ I have added a new database in Redis. Do my changes need to be replicated somewhere else?

DevOps Checklist

[codecov]

Codecov Report

Merging #5043 (ecdaa98) into master (63414c2) will increase coverage by 1.3%.
The diff coverage is 98.6%.

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #5043      +/-   ##
=========================================
+ Coverage    87.2%   88.5%    +1.3%     
=========================================
  Files        1259     775     -484     
  Lines       51679   34724   -16955     
  Branches     1111     700     -411     
=========================================
- Hits        45074   30763   -14311     
+ Misses       6365    3803    -2562     
+ Partials      240     158      -82     

Flag	Coverage Δ
integrationtests	64.9% <81.6%> (+<0.1%)	⬆️
unittests	85.9% <98.6%> (+0.8%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
packages/service-library/src/servicelib/redis.py	93.9% <ø> (-0.3%)	⬇️
...re_service_director_v2/modules/api_keys_manager.py	100.0% <100.0%> (+12.5%)	⬆️
...v2/modules/dynamic_sidecar/docker_compose_specs.py	97.9% <100.0%> (ø)
...c_sidecar/scheduler/_core/_events_user_services.py	86.4% <ø> (ø)
...s/dynamic_sidecar/scheduler/_core/_events_utils.py	94.4% <ø> (ø)
...ector_v2/modules/osparc_variables_substitutions.py	60.9% <100.0%> (ø)
...e_director_v2/utils/base_distributed_identifier.py	97.7% <97.7%> (ø)

... and 490 files with indirect coverage changes

[sanderegg]

please re-use the redis fixture, or modify it but do not duplicate.
Thanks!

[mrnicegyu11]

When this PR goes in, you have to merge ITISFoundation/osparc-ops-environments#445 and make sure redis-commander is redeployed @GitHK

[mrnicegyu11]

ack

[pcrespov]

So this PR is about:

• Extending BaseDistributedIdentifierManager:
  • incorporates a new "cleanup context" concept (name is not very intuitive)
  • connect it to redis
  • changed a bit the previous interface
• Using the above to update the ApiKeysManager with a "Cleanup Context"

Sorry, it gets too complicated for me to follow at this stage :-)Please consider my suggestions. Perhaps we can clarify some concepts offline.
thx anyway.

[codeclimate]

Code Climate has analyzed commit ecdaa98 and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.1% Duplication