Implement and explain extra return codes (kube-burner#688)

## Type of change

- [ ] Refactor
- [x] New feature
- [ ] Bug fix
- [ ] Optimization
- [x] Documentation Update

## Description

Implementing new alert and measurement return codes and add a section to
explain them.

@dry923, regarding the `Create object failure` return code you suggested
in kube-burner#684, this is an error we're not currently tracking, at the moment
kube-burner retries the object creation when it fails and the failure is
different from:

- Object already exists: In this the execution goes on and the error is
printed but ignored
- API authorization error: When this error happens kube-burner stops the
benchmark executionand exits with rc 1

If you're curious, see the logic here
https://github.com/kube-burner/kube-burner/blob/e1880a31d261626ebc210ec24e187a6ac3d03dce/pkg/burner/create.go#L274-L318


## Related Tickets & Documents

- Related Issue kube-burner#684
- Closes #

Signed-off-by: Raul Sevilla <[email protected]>

docs/cli/index.md

@@ -80,6 +80,18 @@ A metrics-endpoints.yaml file with valid keys for the `init` command would look
   alerts: [alert-profile.yaml]
 ```
 
+### Exit codes
+
+Kube-burner has defined a series of exit codes that can help to programmatically identify a benchmark execution error.
+
+| Exit code | Meaning |
+|--------|--------|
+| 0 | Benchmark execution finished normally |
+| 1 | Generic exit code, returned on a unrecoverable error (i.e: API Authorization error or config parsing error) |
+| 2 | Benchmark timeout, returned when kube-burner's execution time exceeds the value passed in the `--timeout` flag |
+| 3 | Alerting error, returned when a `error` or `critical` level alert is fired |
+| 4 | Measurement error, returned on some measurements error conditions, like `thresholds` |
+
 ## Index
 
 This subcommand can be used to collect and index the metrics from a given time range. The time range is given by:

docs/observability/indexing.md

@@ -101,7 +101,7 @@ This document looks like:
   "elapsedTime": 8.768932955,
   "version": "v1.10.0",
   "passed": true,
-  "executionErrors": "this is an exmample",
+  "executionErrors": "this is an example",
   "jobConfig": {                          
     "jobIterations": 1,                                                                                              
     "name": "cluster-density-v2",                                                                                    

pkg/alerting/alert_manager.go

@@ -20,6 +20,7 @@ import (
 	"fmt"
 	"io"
 	"math"
+	"os"
 	"path"
 	"strings"
 	"text/template"
@@ -41,6 +42,7 @@ const (
 	sevError        severityLevel = "error"
 	sevCritical     severityLevel = "critical"
 	alertMetricName               = "alert"
+	rcAlert                       = 3
 )
 
 // alertProfile expression list
@@ -211,7 +213,8 @@ func parseMatrix(value model.Value, description string, severity severityLevel,
 			case sevError:
 				errs = append(errs, errors.New(msg))
 			case sevCritical:
-				log.Fatalf("🚨 %s", msg)
+				log.Errorf("🚨 %s", msg)
+				os.Exit(rcAlert)
 			default:
 				log.Infof("🚨 %s", msg)
 			}

pkg/burner/job.go

@@ -74,6 +74,8 @@ const (
 	jobIteration         = "Iteration"
 	jobUUID              = "UUID"
 	rcTimeout            = 2
+	rcAlert              = 3
+	rcMeasurement        = 4
 	garbageCollectionJob = "garbage-collection"
 )
 
@@ -208,7 +210,7 @@ func Run(configSpec config.Spec, kubeClientProvider *config.KubeClientProvider,
 			if err = measurements.Stop(); err != nil {
 				errs = append(errs, err)
 				log.Error(err.Error())
-				innerRC = 1
+				innerRC = rcMeasurement
 			}
 			if !job.SkipIndexing && len(metricsScraper.IndexerList) > 0 {
 				msWg.Add(1)
@@ -254,7 +256,7 @@ func Run(configSpec config.Spec, kubeClientProvider *config.KubeClientProvider,
 				if err := alertM.Evaluate(job); err != nil {
 					errs = append(errs, err)
 					jobAlerts = append(jobAlerts, err)
-					innerRC = 1
+					innerRC = rcAlert
 				}
 			}
 			if len(jobAlerts) > 0 {

pkg/prometheus/prometheus.go

@@ -48,7 +48,7 @@ func NewPrometheusClient(configSpec config.Spec, url string, auth Auth, step tim
 	return &p, err
 }
 
-// ScrapeJobsMetrics gets all prometheus metrics required and handles them
+// ScrapeJobsMetrics fetches and indexes the configured prometheus expressions
 func (p *Prometheus) ScrapeJobsMetrics(jobList ...Job) error {
 	if p.indexer == nil {
 		log.Info("Indexer not configured, skipping metric scraping")