Better scoped metrics

[brayniac]

In #108 we added support to expose metrics for interrupts per NUMA node. There has also been interest in exposing metrics for individual disks, network interfaces, etc.

To better support the various scopes on which we may gather and summarize metrics, we should consider a more holistic approach. For example, it may be beneficial to make a more robust type of Statistic that can hold information about its scope for reporting.

We need to decide if we wish to allow fine-grained control of scoped metrics, or if we would be okay with on/off per-sampler for scoped metrics. If we need fine-grained control, a more robust scheme for serializing/deserializing, particularly with regard to specifying the metrics in the configuration, will be necessary.

[kash203]

Hi @brayniac - san.
I'm interesting in getting individual metrics for example cpu/frequency of each core, and other interrupts/${nic} of each core because my environment has a lot of cores, interfaces.

I think to achieve getting above, each enum string need to be variable in XxxStatistics.
For example,

impl Statistic<AtomicU64, AtomicU32> for CpuStatistic {
    fn name(&self) -> String {
        match self {
            CpuStatistic::UsageUser(core) => format!("cpu/usage/user/{}",core),
            CpuStatistic::UsageNice(core) => format!("cpu/usage/nice/{}",core),
[...]

The relevant part is below.

rezolus/src/samplers/cpu/stat.rs

Line 100 in 41d4b1d

fn name(&self) -> &str {

(I assume the core information can get from HardwareInfo. It need a little change)

In the current implementation, we must need to know the number of cores, numa nodes and name of interfaces
however those information is different between environments,
so I think that it is not realistic to hard-code the target device in advance.

To make string of enum be variable, name() need to return as String instead of &str.
But this fuction is in trait of rustcommon-metrics, so we cannot change it in Rezolus.
I think the current implementation is kind for memory utilization however to achieve getting individual metrics, I think this method is not bad.

I think you are worried about various things, so please let me know if you have any problems.