CMS: probe checking the number and volume of rules per rse, activity and state

[eachristgr]

Related to dmwm/CMSRucio#747

The script acts on rules with the following states:

• OK
• REPLICATING
• STUCK

The script pushes the following metrics by using PrometheusPusher:

• rule_count_per_rse_activity_state: The number of rules per rse, activity and state
• rule_volume_per_rse_activity_state: The total size (in bytes) of rules per rse, activity and state

Tested it on Integration and it works as far as I can see.
More details about the implementation are commented in the script.

FYI @haozturk

[eachristgr]

Changes were applied in favor of readability and commits were squashed to ensure a cleaner merge.

[ericvaandering]

Sorry, two more changes.

1. Please put a newline at the very end
2. The commit has to be renamed to designate CMS: "The subject of each commit must be prefixed by a label. This can be omitted for rare changes across the entire repository. If there is an issue related to the commit, then its number should be appended. For example, a commit may resemble this: ATLAS: Port probes to Python 3 #1234" per the Contributing.md file

[eachristgr]

Sorry, two more changes.

1. Please put a newline at the very end

2. The commit has to be renamed to designate CMS: "The subject of each commit must be prefixed by a label. This can be omitted for rare changes across the entire repository. If there is an issue related to the commit, then its number should be appended. For example, a commit may resemble this: ATLAS: Port probes to Python 3 #1234" per the Contributing.md file

Changes applied. Hope the commit message looks good as well.

[ericvaandering]

Thanks!