From 3471561346d0ea9bdf45588a5e4ecd79ceb4bb5f Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Fri, 19 Mar 2021 11:59:51 +0100 Subject: [PATCH 01/17] docs: Move architecture diagrams into sub directory Move all `.uml` to a `diagrams` directory, so the directory only contains architecture document which is cleaner. --- docs/developer/architecture/deployment.rst | 2 +- .../architecture/{ => diagrams}/deployment.uml | 0 .../{ => diagrams}/logs-components-diagram.uml | 0 .../{ => diagrams}/solutions-interaction.uml | 0 .../{ => diagrams}/volume-creation_seqdiag.uml | 0 .../{ => diagrams}/volume-deletion_decision_tree.uml | 0 .../{ => diagrams}/volume-deletion_seqdiag.uml | 0 .../volume-deploy_volume_flowchart.uml | 0 .../volume-finalize_volume_flowchart.uml | 0 .../{ => diagrams}/volume-main_loop_flowchart.uml | 0 docs/developer/architecture/logs.rst | 2 +- docs/developer/architecture/solutions.rst | 2 +- docs/developer/architecture/volume.rst | 12 ++++++------ docs/developer/solutions/introduction.rst | 2 +- 14 files changed, 10 insertions(+), 10 deletions(-) rename docs/developer/architecture/{ => diagrams}/deployment.uml (100%) rename docs/developer/architecture/{ => diagrams}/logs-components-diagram.uml (100%) rename docs/developer/architecture/{ => diagrams}/solutions-interaction.uml (100%) rename docs/developer/architecture/{ => diagrams}/volume-creation_seqdiag.uml (100%) rename docs/developer/architecture/{ => diagrams}/volume-deletion_decision_tree.uml (100%) rename docs/developer/architecture/{ => diagrams}/volume-deletion_seqdiag.uml (100%) rename docs/developer/architecture/{ => diagrams}/volume-deploy_volume_flowchart.uml (100%) rename docs/developer/architecture/{ => diagrams}/volume-finalize_volume_flowchart.uml (100%) rename docs/developer/architecture/{ => diagrams}/volume-main_loop_flowchart.uml (100%) diff --git a/docs/developer/architecture/deployment.rst b/docs/developer/architecture/deployment.rst index c8dd4ab724..aff18d5a8f 100644 --- a/docs/developer/architecture/deployment.rst +++ b/docs/developer/architecture/deployment.rst @@ -4,7 +4,7 @@ Deployment Here is a diagram representing how MetalK8s orchestrates deployment on a set of machines: -.. uml:: deployment.uml +.. uml:: diagrams/deployment.uml Some notes ---------- diff --git a/docs/developer/architecture/deployment.uml b/docs/developer/architecture/diagrams/deployment.uml similarity index 100% rename from docs/developer/architecture/deployment.uml rename to docs/developer/architecture/diagrams/deployment.uml diff --git a/docs/developer/architecture/logs-components-diagram.uml b/docs/developer/architecture/diagrams/logs-components-diagram.uml similarity index 100% rename from docs/developer/architecture/logs-components-diagram.uml rename to docs/developer/architecture/diagrams/logs-components-diagram.uml diff --git a/docs/developer/architecture/solutions-interaction.uml b/docs/developer/architecture/diagrams/solutions-interaction.uml similarity index 100% rename from docs/developer/architecture/solutions-interaction.uml rename to docs/developer/architecture/diagrams/solutions-interaction.uml diff --git a/docs/developer/architecture/volume-creation_seqdiag.uml b/docs/developer/architecture/diagrams/volume-creation_seqdiag.uml similarity index 100% rename from docs/developer/architecture/volume-creation_seqdiag.uml rename to docs/developer/architecture/diagrams/volume-creation_seqdiag.uml diff --git a/docs/developer/architecture/volume-deletion_decision_tree.uml b/docs/developer/architecture/diagrams/volume-deletion_decision_tree.uml similarity index 100% rename from docs/developer/architecture/volume-deletion_decision_tree.uml rename to docs/developer/architecture/diagrams/volume-deletion_decision_tree.uml diff --git a/docs/developer/architecture/volume-deletion_seqdiag.uml b/docs/developer/architecture/diagrams/volume-deletion_seqdiag.uml similarity index 100% rename from docs/developer/architecture/volume-deletion_seqdiag.uml rename to docs/developer/architecture/diagrams/volume-deletion_seqdiag.uml diff --git a/docs/developer/architecture/volume-deploy_volume_flowchart.uml b/docs/developer/architecture/diagrams/volume-deploy_volume_flowchart.uml similarity index 100% rename from docs/developer/architecture/volume-deploy_volume_flowchart.uml rename to docs/developer/architecture/diagrams/volume-deploy_volume_flowchart.uml diff --git a/docs/developer/architecture/volume-finalize_volume_flowchart.uml b/docs/developer/architecture/diagrams/volume-finalize_volume_flowchart.uml similarity index 100% rename from docs/developer/architecture/volume-finalize_volume_flowchart.uml rename to docs/developer/architecture/diagrams/volume-finalize_volume_flowchart.uml diff --git a/docs/developer/architecture/volume-main_loop_flowchart.uml b/docs/developer/architecture/diagrams/volume-main_loop_flowchart.uml similarity index 100% rename from docs/developer/architecture/volume-main_loop_flowchart.uml rename to docs/developer/architecture/diagrams/volume-main_loop_flowchart.uml diff --git a/docs/developer/architecture/logs.rst b/docs/developer/architecture/logs.rst index bca0e94eac..54601942de 100644 --- a/docs/developer/architecture/logs.rst +++ b/docs/developer/architecture/logs.rst @@ -202,7 +202,7 @@ Components Our Log Centralization system can be split into several components as follows: -.. uml:: logs-components-diagram.uml +.. uml:: diagrams/logs-components-diagram.uml Collector ~~~~~~~~~ diff --git a/docs/developer/architecture/solutions.rst b/docs/developer/architecture/solutions.rst index 18dad1ab6c..1147cc8fbd 100644 --- a/docs/developer/architecture/solutions.rst +++ b/docs/developer/architecture/solutions.rst @@ -308,7 +308,7 @@ Interaction diagram We include a detailed interaction sequence diagram for describing how MetalK8s will handle user input when deploying / upgrading Solutions. -.. uml:: solutions-interaction.uml +.. uml:: diagrams/solutions-interaction.uml Rejected design choices ----------------------- diff --git a/docs/developer/architecture/volume.rst b/docs/developer/architecture/volume.rst index f9461fa463..ff5e2731ab 100644 --- a/docs/developer/architecture/volume.rst +++ b/docs/developer/architecture/volume.rst @@ -116,9 +116,9 @@ system) through the Salt API. Authentication to the Salt API will be done though a dedicated Salt account (with limited privileges) using credentials from a dedicated cluster **Service Account**. -.. uml:: volume-creation_seqdiag.uml +.. uml:: diagrams/volume-creation_seqdiag.uml -.. uml:: volume-deletion_seqdiag.uml +.. uml:: diagrams/volume-deletion_seqdiag.uml Implementation Details @@ -198,7 +198,7 @@ Once pre-checks are done, there are four cases: 4. the backing **PersistentVolume** exists: the operator will check its status to update the volume's status accordingly. -.. uml:: volume-main_loop_flowchart.uml +.. uml:: diagrams/volume-main_loop_flowchart.uml .. _volume-deployment: @@ -232,7 +232,7 @@ Once the **PersistentVolume** is successfuly created, the operator will move the **Volume** to the `Available` state and reschedule the request (the next iteration will check the health of the **PersistentVolume** just created). -.. uml:: volume-deploy_volume_flowchart.uml +.. uml:: diagrams/volume-deploy_volume_flowchart.uml Steady state ~~~~~~~~~~~~ @@ -262,7 +262,7 @@ becomes unused (this is done by rescheduling). Once the backing **PersistentVolume** becomes unused, the operator will reclaim its storage and remove the finalizers to let the object be deleted. -.. uml:: volume-finalize_volume_flowchart.uml +.. uml:: diagrams/volume-finalize_volume_flowchart.uml Volume Deletion Criteria @@ -291,7 +291,7 @@ In the end, a **Volume** can be deleted in two cases: - the backing **PersistentVolume** is not bound (**Available**, **Released** or **Failed**) -.. uml:: volume-deletion_decision_tree.uml +.. uml:: diagrams/volume-deletion_decision_tree.uml Documentation diff --git a/docs/developer/solutions/introduction.rst b/docs/developer/solutions/introduction.rst index 86e5595aad..00e48d7102 100644 --- a/docs/developer/solutions/introduction.rst +++ b/docs/developer/solutions/introduction.rst @@ -123,4 +123,4 @@ will handle user input when deploying / upgrading Solutions. .. note:: Open the image in a new tab to see it in full resolution. -.. uml:: ../architecture/solutions-interaction.uml +.. uml:: ../architecture/diagrams/solutions-interaction.uml From 19325d571fafed5ef5227e5bd43c97894a1bac8d Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Fri, 19 Mar 2021 12:06:16 +0100 Subject: [PATCH 02/17] docs: Alert history design document This document describes how the alert history feature works and explain why. Refs: #3180 --- docs/developer/architecture/alert-history.rst | 285 ++++++++++++++++++ .../diagrams/alert-history_containers.uml | 62 ++++ .../architecture/img/GlobalHealthCpnt.png | Bin 0 -> 28690 bytes .../architecture/models/alert-history.dsl | 39 +++ 4 files changed, 386 insertions(+) create mode 100644 docs/developer/architecture/alert-history.rst create mode 100644 docs/developer/architecture/diagrams/alert-history_containers.uml create mode 100644 docs/developer/architecture/img/GlobalHealthCpnt.png create mode 100644 docs/developer/architecture/models/alert-history.dsl diff --git a/docs/developer/architecture/alert-history.rst b/docs/developer/architecture/alert-history.rst new file mode 100644 index 0000000000..730f8c5357 --- /dev/null +++ b/docs/developer/architecture/alert-history.rst @@ -0,0 +1,285 @@ +Alert History +============= + +Context +------- +In NextGen UI we are introducing the Global Health Component that shows real +time entity Health but also intends to show entity Health over the last X days. +This Global Health Component is available in the System Health Monitor as well +as in Metalk8s, xCore and XDM admin UIs. It applies to entities like Node, +Volume, Platform, Storage Backends, etc. + + .. image:: img/GlobalHealthCpnt.png + +The entity Health is computed based on active alerts. In order to know the +health of an entity as it was in the past, we would need to collect alerts that +were active for this specific past time. As soon as the Platform or Storage +Admin identifies a time at which the entity was degraded, he can access the +detailed list of sub alerts impacting this entity. + +Currently, once an active alert is cleared, it disappears from the System. + +Goal +---- +In order to achieve the UI functionality as described above, we would need to +keep information about the alerts that were fired in the past: + +- The alert (all info that were available at the time the alert was fired) +- When it was fired +- When it was cleared + +User Stories +------------ +As a Platform/Storage Admin, I want to know the health of a given NextGen +entity over the past X days in order to ease root cause analysis. + +Basically this should be achieved by collecting past alerts, belonging to this +entity. + +As a Platform/Storage Admin, I want to collect the list of sub alerts which +contributed to the degradation on a specific entity in the past, in order to +understand more in details the cause of the degradation. + +Here we would need to access all sub alerts (contributing to the entity high +level alert). This is related to the Alert grouping feature. + +The X days to keep accessible is configurable and ideally matches with the +history of other observability data (metrics and logs) in order to ease the +correlation between various observability indicators. +This configuration must be persistent across platform upgrades. + +In conclusion, the system should retain all emitted alerts for a given +configurable period. + +The service exposing past alerts is to be used by NextGen Admin UIs. It can +also be used by some NextGen tooling when it comes to create a support ticket. +It will not be used by xCore or XDM data workloads and it will not be exposed +for external usage. + +Monitoring and Alerting +----------------------- +The service exposing past alerts should be monitored i.e. should expose key +health/performance indicators that one can consume through dedicated +Grafana dashboard. An alert should be triggered when the service is degraded. + +Deployment +---------- +The said service is part of the infra service category and it is either +deployed automatically or some documentation explains how to deploy it and +provision storage for it. + +It should support one node failure when deploying NextGen on more than 3 nodes, +like for monitoring, alerting and logging services. + +Future/Bonus Features +--------------------- +Dedicated Grafana Dashboard to navigate through the past alerts without +focusing on a specific entity only. From this dashboard, one can select one or +multiple labels as well as a specific period, in order to collect all alerts +with a given set of labels. + +A dump of the past alerts could be added to the sos report that one would +generate when collecting all information to send to Scality support. + +Design Choices +-------------- + +.. uml:: diagrams/alert-history_containers.uml + +Alertmanager webhook +~~~~~~~~~~~~~~~~~~~~ + +To retrieve alerts sent by Alertmanager, we configure a specific receiver +where it sends each and every incoming alerts. +This receiver is a `webhook`_ which is basically an HTTP server listening +on a port and waiting for HTTP POST request from Alertmanager. +It then forwards alerts to the storage backend. + +Alerts sent by Alertmanager are JSON formatted as follows:: + + { + "version": "4", + "groupKey": , // key identifying the group of alerts (e.g. to deduplicate) + "truncatedAlerts": , // how many alerts have been truncated due to "max_alerts" + "status": "", + "receiver": , + "groupLabels": , + "commonLabels": , + "commonAnnotations": , + "externalURL": , // backlink to the Alertmanager. + "alerts": [ + { + "status": "", + "labels": , + "annotations": , + "startsAt": "", + "endsAt": "", + "generatorURL": // identifies the entity that caused the alert + }, + ... + ] + } + +Alertmanager implements an exponential backoff retry mechanism, so We can not +miss alerts if the webhook is unreachable/down. +It will keep retrying until it manages to send the alerts. + +.. _webhook: https://prometheus.io/docs/alerting/latest/configuration/#webhook_config + +Loki as storage backend +~~~~~~~~~~~~~~~~~~~~~~~ + +We use Loki as the storage backend for alert history because it provides +several advantages. + +First, it allows to easily store the alerts by simply logging them on the +webhook container output, letting Fluent-bit forward the alerts to it. + +Loki uses a NoSQL database, which is better to store JSON documents than +an SQL one, allowing us to not have to create and maintain a database schema +for the alerts. + +Loki also provides an API allowing us to expose and query these alerts using +the LogQL language. + +Plus, since Loki is already part of the cluster, it saves us from having +to install, manage and expose a new database. + +Using Loki, we also directly benefit from its retention and purge mechanisms, +making the alerts history retention time automatically aligned with all +other logs (14 days by default). + +.. warning:: + + There is a drawback in using Loki, if at some point its volume is full + (because there is too much logs), we will not be able to store new alerts + anymore, especially since there is no size-based purge mechanism. + + Another issue is, since we share the retention configuration with the other + logs, it is hard to ensure we will keep enough alert history. + + As for now, there is no retention based on labels, streams, tenant or + whatever (on-going discussion `GH Loki #162`_). + +.. _GH Loki #162: https://github.com/grafana/loki/issues/162 + +Rejected Design Choices +----------------------- + +Alertmanager API scraper +~~~~~~~~~~~~~~~~~~~~~~~~ + +A program polling the Alertmanager API to retrieve alerts. + +It generates more load and forces us to parse the result from the API +to keep track of what we already forward to the storage backend +or query it to retrieve the previously sent alerts. + +Plus, it does not allow to have alerts in near to real time, +except if we poll the API in a really aggressive manner. + +If the scraper is down for a long period of time, we could also +loose some alerts. + +Dedicated database as storage backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using a dedicated database to store alerts history was rejected, because +it means adding an extra component to the stack. + +Furthermore, we would need to handle the database replication, lifecycle, etc. + +We would also need to expose this database to the various components consuming +the data, probably through an API, bringing another extra component to +develop and maintain. + +Implementation Details +---------------------- + +Alertmanager webhook +~~~~~~~~~~~~~~~~~~~~ + +We need a simple container, with a basic HTTP server running inside, +simply handling POST requests and logging them on the standard output. + +It will be deployed by Salt as part of the monitoring stack. + +A deployment with only 1 replica will be used as we do not want +duplicated entries and Alertmanager handles retry mechanism if the webhook +is unreachable. + +An example of what we need can be found +`here `. + +Alertmanager configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The default Alertmanager's configuration must be updated to send all +alerts to this webhook. + +Configuration example:: + + receivers: + - name: metalk8s-alert-logger + webhook_configs: + - send_resolved: true + url: http://: + route: + receiver: metalk8s-alert-logger + routes: + - receiver: metalk8s-alert-logger + continue: True + +This configuration must not be overwritable by any user customization +and the ``metalk8s-alert-logger`` receiver must be the first route to +ensure it will receive all the alerts. + +Fluent-bit configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +Logs from the webhook need to be handled differently than the other Kubernetes +containers. +Timestamps of the logs must be extracted from the JSON ``timestamp`` key and +only the JSON part of the log must be stored to make it easier to use by +automatic tools. + +Expose Loki API +~~~~~~~~~~~~~~~ + +The Loki API must be reachable via the web UI, therefore it must be exposed +through an ingress as it is already done for Prometheus or Alertmanager APIs. + +Grafana dashboard +~~~~~~~~~~~~~~~~~ + +Alerts are already retrievable from the ``Logs`` dashboard, but it is not +user friendly as the webhook pod name must be known by the user and +metrics displayed are relative to the pod, not the alerts themselves. + +A dedicated Grafana dashboard with the alerts and metrics related to them will +be added. + +.. todo:: + + Information/metrics to display need to be defined + +This dashboard will be deployed by adding a ConfigMap +``alert-history-dashboard`` in Namespace ``metalk8s-monitoring``: + +.. code-block:: yaml + + apiVersion: v1 + kind: ConfigMap + metadata: + name: alert-history-dashboard + namespace: metalk8s-monitoring + labels: + grafana_dashboard: "1" + data: + alert-history.json: + +Test Plan +--------- + +Add a test in post-install to ensure we can at least retrieve the ``Watchdog`` +alert using Loki API. diff --git a/docs/developer/architecture/diagrams/alert-history_containers.uml b/docs/developer/architecture/diagrams/alert-history_containers.uml new file mode 100644 index 0000000000..3b23ad5162 --- /dev/null +++ b/docs/developer/architecture/diagrams/alert-history_containers.uml @@ -0,0 +1,62 @@ +@startuml(id=Container) +title Alert history - Containers +caption A detailed view of the alert history system. + +skinparam { + shadowing false + arrowFontSize 10 + defaultTextAlignment center + wrapWidth 200 + maxMessageSize 100 +} +hide stereotype +skinparam rectangle<<1>> { + BackgroundColor #08427b + FontColor #ffffff + BorderColor #dddddd +} +skinparam rectangle<<2>> { + BackgroundColor #1168bd + FontColor #ffffff + BorderColor #dddddd + roundCorner 20 +} +skinparam rectangle<<6>> { + BackgroundColor #438dd5 + FontColor #ffffff + BorderColor #dddddd + roundCorner 20 +} +skinparam rectangle<<7>> { + BackgroundColor #438dd5 + FontColor #ffffff + BorderColor #dddddd + roundCorner 20 +} +skinparam rectangle<<8>> { + BackgroundColor #438dd5 + FontColor #ffffff + BorderColor #dddddd + roundCorner 20 +} +skinparam rectangle<<9>> { + BackgroundColor #438dd5 + FontColor #ffffff + BorderColor #dddddd + roundCorner 20 +} +rectangle "==MetalK8s Logging\n[Software System]" <<2>> as 2 +rectangle "==User\n[Person]\n\nA MetalK8s user." <<1>> as 1 +package "MetalK8s Monitoring\n[Software System]" { + rectangle "==Alert Logger\n[Container]\n\nService logging alerts received from Alertmanager." <<6>> as 6 + rectangle "==Alertmanager\n[Container]\n\nService deduplicating, grouping and forwarding alerts." <<7>> as 7 + rectangle "==Grafana\n[Container]\n\nService displaying metrics and logs through dashboards." <<9>> as 9 + rectangle "==Prometheus\n[Container]\n\nService monitoring the cluster components." <<8>> as 8 +} +7 .[#707070,thickness=2].> 6 : "Forwards alerts to" +9 .[#707070,thickness=2].> 2 : "Queries logs (alerts) from" +9 .[#707070,thickness=2].> 8 : "Queries metrics from" +2 .[#707070,thickness=2].> 6 : "Reads logs from" +8 .[#707070,thickness=2].> 7 : "Sends alerts to" +1 .[#707070,thickness=2].> 9 : "Consults alerts dashboard" +@enduml \ No newline at end of file diff --git a/docs/developer/architecture/img/GlobalHealthCpnt.png b/docs/developer/architecture/img/GlobalHealthCpnt.png new file mode 100644 index 0000000000000000000000000000000000000000..9571e109486d9fba06e02578ea55a473d733f5a0 GIT binary patch literal 28690 zcmeFX^Lw357d9L=cEiTDZCed@(%80b+fLfpw$qr6?Ivk#>)UfcHAC zYtQU8v(|X#Tx(6ZqPzqGEDkIP2nd3dq^J@I2pBmqegpjh_*-3kDgy!nZ*DFkq9`RI zLa69qYhrF?3<4qI8OI14$xIk|3oz{Y;V*g>Vdoh0Jr^O7t1|lMpWX zWOTUNAq%sZK5p2yg-~($6<0X&u}Fs`A)K7zvQ>(FjQXSSF>|3+inMPE4cmji~=7GGJ}cbqhQacFv-O!j{~!R+C%Mxw(6xq8mP?>#tO2tdoW^l z5t6E;^W13>%1bL=-bh~s?j}3hN04&J8^>nCIebU=a12{FhJ-0=b#vg?%98fsRZbvC zEr~*77~dBkq}uIG4K(ZdK;#GWv&PtOgz!_CX8Te5mcX7Q))pAmFc{cY$qf9YBsLbx zcS|rz2w!1Rkh-t#z2Ny^iXgYTAq}JDB*VB^bXaNe;5ousP~QaUUl<6|$ndXH;By3{ ze-Ui7)?Dg!wAN4!zIXa4KFfU%$S_M5=l>O7%4H@6PZokq5~eT8vxbb+^LkYIlVYozqx@Z=`#9g_I0(+{eeHdHeP=ne=O{oBGkQ-q-{edkA^FL-9G^~LbF?(` z=C&i?K>5j7x8+$t$a}z+`nL5$SeZ&w+hTfuQTnVs zht()9@oj?=E?O9E8paHR8mT)Kxfb zw$MPRi9?s$l%yudBjhP~JfCODX>P~*faAzY&343EW!5o6YN=^n$4X&=VUc2q|D&T| zxlnAOZo$CPzR|=|)dFv!t#m$LZqa2rW`1&3=Ub=DbJm?BUP4Txd%-Mw#lP^{M72L@$JjL4l%C2LXAO%O(mZ4_SW-25c}GszS(qak6Em%uEZL4(nwb=s z8JIUMeKrFOwyopMKbuS&Ok0~1@P>!MVW6?_|I!#)Hl^&(n!x&c$r~Ga}wp+{gLOZqj*U-)m89h-{{C zxNv$ey1bKSs&VEvo*IxG1odf-V32yXeTT~#Uls#)y1}5=RTX=FSxG`FD{RK1ja1Vj25dULvQ5<(b4FL|dzQvjC=s|A~da=imR^NX)rNLIRcv@VX& z7^wJ4s53ZaXc-^d=x>D5+yQC;Ie=@6#6#4@KD4l?N`FLHCaxi3tn$QR%uPfVx;t|O zPBL;M{n6(J?*`U3dsc34hnuqLdxHvtex}=6!W6W2>N}q-xwWL0%|Ya${^(NNYt{`i zACs4K2#>ykAZ?g)v_P~}vJJU9xde(vQcLn!g(~^CIMjqX=0fk2;oAB^9Z5D>^g{N6 z_?d*+rsCTuIJ{;Lts%GA%Yj{6*_w1l>%P|T5Hv>${PY5@aGpa!ZX!N1Z}lfr8`|cG z)x_QrK<@N};q}6W&Q;)p@r%5Fw_f3vMv?FS==g!gn?`vZcd&n-gMMllohF4xHeP%u z)w77a)nrW8+gj3Er@Mvwp?l~6_YUsm$Ow62v7AjJr(^`#sj^pbMa~Rebb?JPQ5{TO zh?n-mXW!2@H1ASxHH3M2b$PB9S99RA4aMQb+F3RN-j_F~!TQ17<2|MqCJp1PnOwH} z-!&>UzL=2IvzRoG*QV9$$q%hvwDKH$-rpO4?i_A4GoEgLq`LluiN%oCzICavEinJl z@}o?oE&a`x7ITy?$%Yc=+SQ%L$nKO2naNb{)Ra^ooxU#)Z>FiaG(Km#8LbV@=?8Vu zW8q^1W40C0TD6XaccXpDvFgz}s_KKZG*q6=9y3RdduMx@3yKTFiw)|c>Z}!wTJ2t^ z2hM%Yi!^~W_f;I4-EOa;2rmfg>>sQv){Qz!oo0(CzD+bY1=*IC*Huw(O0Q|w+#WXl zv^Mp*d@J;5-@cnU)pZGTZ8(qGbgx?}Z}Ui0M zb|P4!*u39aywSBiiCx)r=y?`A6)F!rMw&)So+EOeJ#IS=zCA@KMX(MfL271wV#9x$ zIBdp~Gm%S{xQU_Vk>+aKGJCJ>Gvu2{*st2v7$RtvwP^J;I0?6;7|OEZLw7zpIjotb zn5>z^4oru` zi^cH#>>}qPy;f}Nh6~BzU}fg6ZPVt?riTyD{oOGSr0dh!WT!-nx`Xbq&m(v~l&bI2 zYi;v-{p;}exKK<2GTyf5wEKrWwB^JpJh+^hoD=~TZ?)I8TX$tsz`-8ou8vbjQpd?L z^_%kZ)NQquF5U~xujCo9t#||EQgK*nC%SFy7^rj$5Y4Wsdj~H{tBS*?5r$kejGd?d4*ttp$F`voO>E} zg2;Q1&LIBpLCFULpqCJN7%mZ0ZZ0e!&wBMaaIN~vUVRiFK<6f`OG-sk4)@D`e(Os_ z{@UFAUdaSfAP&+c`TYFMesk};{2{OtLGb$H9<~aRsGG?W?l-|^_2^|VAE5Y~c3j^R ze_}A#3&VGY)uA}@%1R_CpzaZ243ILBlLMg!#-TwVKyg4IfiY0v#}E4HZ(JOd5(NA| zbTAN*P;(H7KY8SV;qNC7`2FVjCj?Ij0r@Q(75KU3f&H5sj64tg-#Ew(FbzaVSwu<- z7%Ce&7#rI-n%O!zNCwjY2{3k&8jc_!7-YXcP$?zS3!wZ3a}|ISKu(t1(AJt>-^kX$ znBL9W?zbEeUN>%F)Y{ldpU}BQ|q#vg2l8aCLR1cV(fsbueXM;^N|BU<96D z%yhsUbdK&ePWoDlzsIjA=gSnlPxvdT1Z@Kygw$4s`#KgY~`uqG-Ph&Up z|CVIq_@`OG1~U9kVPK+XWcVu^$jbYh%B^VbW^AP)YHkheGhi9~%&e@u|H=PM_E6~@u$4?&Lv>?1k52LPA)PqA*?z`j)Q?aLzgLs$v}2@a3O#|V}FQ)-U9Utj-h zF5`j3|Eq=|*#Dp2sHtR>N~O;>f`?9$O=pqO(~Dc{^8P}v+eMm6uamS?t`31FVsC#UQ|cgV zQ%{ztRhTThy86z3+e>!l5dQ6LB_uVqIg8)@C5D~Zc+y}$L2PHoW;k7$!kNDX+u8(X zk{l~AHdaB&)6MaD!J1_g&SC#%#Ax#x{NE8z_B#y)v{*g1vEFWUydUkrzn=x1!K8E-EnA0A(WJb@N^eU|s_a!?$T)U)5+;kM;)FCC zPxE#Rd$TmC5pj_k?WT9Gj4HVu=-AbK(wIM2>9p=}CZC}1QkzcYEWF`i=EaUG@RCu);w6h+`^ciIyhZp+3Wi8Sv{H>_zf?LGJn2SMETCYt9G zm^`08a=APx&UhD&s>Zb;WMa{bvZY@YPy$5w%7G0Ho!q3{u>m za8lHBtmic^+B^>sHCSDnvN(Nc54{er(4!wp>4su)Q?QlEN9ah(>>xfEC*U|ANi9bt zPk70XNIAxS#|c^te7+w!&txZU{Lb<3>NN#xLM7kW;7?cwNNYBon$aTp*O|ZKEZ6x| zwh44}bVSjRmeZfb$MTbVj_-8qozb=;U+cO`e`BPoK0|4dDU!!!IEbnV=cvMVg-PUf zzfQj#9BES>Q*F`W_0@UzITKH-&~DXQNcY8Jh*{kiaP!?AM0g#4Cjvz!{Qb70)g&|e ze*sSk^gzfQJ=69bp1M9rbBjik!hmRTBZlRpo?^A8_Zh5Xeq62odC?oV@Ad5#w6DM} ze#ha}qhiqUWTR0zh4x8G{!{!;ZtuJQ7ug@aO($G(?~5vmR+`P?3)MR2K?BbIs^oN$ zBUie^2X;YU8`gq&FIdpsy?|um&xk$;eJ~%?hGC6A~JM15JJ-L_si?=R~aOQ zn5a6t)^Gk3j)4%Z`%UXE6AQ(tmXjd8-XwYD_LKCX1G;g1ZHX12UMWz`9wBIiWE%yH z$l+xe#l7oxY{$4ux6`0hqmj?k;OA1dK2%<-2GD zPe;raDT@W`yi-0!eS-kUA+hKg_{<}PMODm6{zNIKnU;cxWZL91#^r9)SQ9+5w96gN zmqnl55borUl*i-#=TiUaBgvi^2rDvKFxmBlq1+0aI+Y_!@=*qX302n+a9)d`pA5)E zayhnz*gR<6IF<&oh@`FDxP3-@GXr(Y5EI0HO-IUXg`$UbID$AyREY}&>DYXa{ziKx z1ERqr5*7n6BY4z}pIA{L`oA^7GTUt})Q*n&!Np31CviC%#B7RP<)7bT~e%S?U{zj%Z#_#qUHh^{fPX(DME>ktBQ2pqHf6K->oX zKH1Ei`#y!kg;^LCB28Z-u?Ru={-uFH{!{jdv%tqNp2>tIC7y(8 zk%2hK0tHU0ta*?cflH1`45muE>C-rKRbt()R`DhAV_ei7V;tVO;$B(4Oqe%(u z)|xWmr~}c0m+F=65%8bNh7@f@a9nInMoO5y(pUTt`a(9vNm*PEuky*^7u<0zPiOk}WGMjIEj!(u_h5D?goBvQmSrE}P( zX?1ytt+RX&jcpkJMWsln^#T4D1v-tIA``RYz7;n2jd!ox|5;#vyh;T?Cj{Y>0qrNQ zxrwW@_Q0Rbn?4G476qc9mXae0WPACsA?0ik8f={&xA6k+Yn7Lj&@PqQ?I|7C@&YjM z@NsgPO<3iljC@FfRWW&af{iOL&zmBogoJ|ZR!c@fqn-}Geh@`V**hb3ll)!8*+3Kv zQ)iRGVM|`!xuCHtODL)1`-;HjNe)HfT1bDy`PCWf2&d5zE4!XbAlfUsq2;xRH43>=nMQu0!~#3J)0O z^eUa`#tz&|j?!q*-7bf{Q9Nu#Xz>te(G+E^HM!0YiyC-afd~Gcdf{!E?tsRDKZxK& zsK=!bBGRc5^GiC&ylkDzDfi)zul1Vz6`~OMkQlG&EcwyT#bF}cMSc3oi7`fDz& zMxA7DKzv4qR-PbeJqQ%ew38CJ8;8wW0T?V?v!PC!ClJ`O4Gy$c&TX7;bV}@xr&B;9 zHgVTH986}*r0-;iMf#xJoFDo5na`IMexz3UF>n5gXqdxc+jX*3%W{YVL4!#PP|Wc- zf|*sf*`;rX7WNV_xjZ)7*Ri&-Q;u ztS6$MAa)$tV1Atdn>dl#jAc1=-1bAGRhv+{=0R9A59AYUm6(`N+DkzSXaSDv@m+`% zkI$f1st%?@i?JS84fhXnW+#)0j3SIwkx)d-sF5<&^4X8%$ZXJ~*2`D!a;_m_AW#^l z3=Fo>?#pe?nO^tn_jPWU2?Z_a0MDnpQ#u9S&l47aC%zR4KSmu)x~FA8D1VaSP)vIV z)M8NRL8|ZTF`Ha_NTW%ASS;%W#~wnOW0O@cv&9^Abfv#>zlI{|;Oj;T8ihP{r14ma zC0Yy8vU25@?viEscGq)^Fd;zgLiMJe=$2li7FTodrNFBbogu=p)lApx`1N6F@TuZR zD%5{eHopT@;Q8K5j0g%!go2x;x+}Qh)1F()e4<(&UM>OoS*Zf>`SD>l-U7Ra__Y^` zV6@KIZcj^D9lp+LrP;Qe9yScF1!z>v!jw5^LQ%hZ*iANmxX`!Y@;5g4k@r=zpRN)0 z0U0b!?pkF;6lp}YT;^?nk6)FXV_7u{b8T!gQeP>_${cgxt(e> zSuhyh=2aFu==Slbag0T=GM(%!1%ULW`ltz*S=z89H@lZXW!#M=Q?#bDmgi19^b-j6X z6cK#S-YuI53j-tfYYMezQJE9Av6H$vf3J$brWr$c(_+jyn$MlN?4i4QJkTADK&O%8 z9W(Za1hdDoD<_PkWM)<}RP(qy$@lL>X5&QqYV1ttlkax9Gbqj6-%>GYK|2wFUfy7R z;Q0e%Fkr`tp?ghyS4J@i(cAZGq?kS!<_^Bk3x(9{<-s1C*DYxg~oh3Tn{9+`GKimzzM{{RK|{?QFeWI`G4-ik8>gBWLHL{l+D1{O1NU6g;>9ThocB&p>cuO)E3- zvwn)-l}fg!-hS6~?lh2=6uP!Si{=+pmX1$rCC1lJeB7f<5a?V$Go;!WvQni&E`Mbe zepn$Q-pZjP4$D4EV-4wg{~vn+zTEHKRsp>Gj=-~`Mq|!X^6_d<&N{&Zw8v=~{pJ^6WdJ(^_2_@1pj{yvbkw2P{Wi9|ML$MbbMpDZq$ z8Y~J3>4EXKQAjY@fDiQK&};|bi@gZ!WO&0}z{Q9@S*|}s6MKO! zYv{|~^&)L!_nJcuNQ0w|d!TpE)LBumI#F#LCY*&BQV^h<;|!m>+p@ob)MxhbGO%xf zgu}p&8~_BLS$Z+I8LNqA#eY2C?Y8--wp%eR+{@3hX|M2%Qy*y>9*ybtXC9PQ|9~a~btir{SnwB-Hf&c*iL=3g&4HLT6SZXyZ*Z_C-5)(n! zKPF-~M9@`OG+wY4LdDwK5sGq)BG{vN&I&BR3nD(RSp@~f4JJ>WAhO8iVXz+*^t!=pU2385T9OB%WYkJnAFtPet{eR?`kR$R`eQJM%6``Z5^GMvkl zR>rqPlE?YTX$HVV05E;J^ImPx9yfxaTM*P$SqhGXqHj1`_dEuyG+G+w?4cFKj4KS5 zu}R&tIM~@=?EYg3@H0iLX=Lop?`3f4kNG(quRZHpi$b7+-7a$TJIsgB_@ezvUNyf*Wo=I+ALf z?p849K;{br_s0s0$@%ImLa)VHor2IJqgUyEJwczL1UDuhidbzn7&A|dKoki17KjvL zp8j@wyjV{5L2Nv$66;sH)iU!@3=D_)?C^sBCn$OvK#~#~IWksBKC{mY=N<|{ESS0a zyf9=fxFsyJ1H0{;oSqyI`Bo3E>aDaidp$a6VlfbZ^NS#Z&PYqUCdYeH&lYeA7niOT z`9;*iUI=$&^Kgap&wlCoTO+d%Ca?hm$SZ~+*kIGPL7RB*M%&;>rmOW7m9y7Ah_E`H zu86JkA=3lA+FiV5FlXDfMQbHV)F2NhosSpPQjzOaYiO>bKiLQHxor*lw?krNn9rJ+ zJyn%6admxfQEY)vr)_~+W)7RM^ecifwwSh3K1Xuk4^kyjDHTt zvgLdbh60=3Qt}5w2|C!PZNRm@PX3R|(&Gs2R=_9}-!Pg?V>wC}kucETZx(`Jz+Hp9 zIR#n{V!#2^*9x5#5DECL0$3$1%Fb_(InGL@!q8e$YT`I77er}>`ZD}E5t7D+Xq3MB zzI&HPz(eQFA+|`z9l`f9xt@7FUVM)?$r&+ZA)F%&!Qrqe2!JKKM1&74}7l4Pb=aTNa$bV!hH~ zBs!k&fv_)NzjLoi62e3=a(IpQ8zi~mfX1Scm!L+ppwZ4j!iig4&;(G%zk*o?7!q}* zLWRR)HUzVO@2h#fFjoF(UA6EmSTXwg5)sM1N9dvW5#jmn=V z4FY4ZVlB5F!%*TDnLo}z=Okl6qyzoTSTtM=5VG^Zhl2dGaqui67huX=BPZI^@G_= z0hI|uK0y*x5Nt%fK8x6mc9)k1oph*M8W1d#4^#sh$Np6eWP0+0U@(SW9toHt*J(0l zA|{?|T6zOOTt{AU&Cseo^m`G)LHC)I+6PAA^NHV`t-A`yZP+6{`3*okPdOc4WuV~1 zW?nry%1*b~HVh8oad3L0pnMjb4J=nJw_yMHA_Q#p7|)J2A(kS1mC3{rj>6SJZhNCa zxiSk*V4$~ooH+#M7*H#Y-jZgd>ivwvdSdlx*ERLX7ZAJYQC|Mxj@ShfU3>e?-nl0` znAJdKO$-;ILcuPS=vT8)pa(T+v(=;Ax-2CYhus#>sP}kUZAQ*%nVio*J~KUEFnluE ztr;^QJX_Y?Q(Z?s|2dMF5zJz_koEkQ9G$nKu5)Mx#*$ec3~qc>%W?^XQEyG0FSc*c z$iDut6~Cb9Q-J4a8sWM~>0La(_lHv8E`=fWJy5h`tjT)yJdNh=rZxQ3lXai z`Wr;GU1|?{5Xd4>xzezyORXcSIa+ToZtDOljfLjJHC~(qcAJD71zN3jny>6KDdU5D z>^&drR!cA0kO^zZzk=Y`dD`{tu-L2tI&FW$=&OxZxvqaK@E5pX_$0D=&OP`nzg5#p z@)|3z$Gyci6nys6_FQmlgZC_;S;eySE|X(dj<;XZa3al;7W81&3N?~3rlA4>P@>Nz z*j7c#L_u7;nOZ<45!5jV3Tb@|v~OK%Lm>UaN&82A{7C zw6Y4r38GM~OhrhWuI7Qz`|dlEgn1bqzPtt0=!xsunp9-+*9DlP_?EIluxUTvEhA^W zMLht`7P~(WCAUYL0|ZahrvR{8_8L3F2@d16-^MRZ+Zt66FgKK!h*nMlEN}z!8Ue&L z1DO=sdFpyeb#VeR>7*Yx3~rl+GU^j+y{}>U*gWjg&k_?@fZ#vI3^#JnRXDV?*X<9z zZ!F0)>NDIVOOB9K^~Vd=E!Cr6k>MVkV}fx?+kr4O3lw}~7J#lE={qVAD?vad6Hwcp_4bYUNL67?vH+$*StS~^tJ6REa}tjlIw2VO`(!^50xfK9 zH1tR1M<-P;*B9#yL4cLJUz-d;%ZdACsmMSTtAf7Mrgl7F4a7?8gh0`K1H;`tDJM#+A#dCMC5SFU|1$PrbrxwT7O`8J)G7P>RVI zLiy$zZiHrW7Eu74ag>E{peeD?J-7UkyGuqUO|K%G+Zos@>g{h;!|5!ha}$AB7yz~& zy1`V4hT3@#UO_sdmz)8~9~ zXY`1rOek7qO-o$IKQ0WvSC}Quw4MZi-No1E`_Gu*0qmFzI^Pbf09bh4{pWj3Unh#> zGRTQsBqhD>~8z<0j+ytZiyZ@6)e5BhA5 zcDP^9%k8ub9&;%h! z0Mv+QTksoV2~DER_eND+(N}kQZT@U2HJCulDq26SproGv&7C~_s3^9}3#{u~qYpzY zg!MNqZUb4pd62x8mQR|=w>m4LpOD#W$n_irUPfKJHd_g13uCZ*i(p_%hm!T^u`t1e zw`c6f?|nqmSTLcH#<8oH9)2t*Y_}l3ev^j>bJ8ag(t>>ohVJLXg45B4A#H$T`arMe zKc}Z|C{EBqZYb_+3?*L!*v7xGcaAmm^7_XQQ6m8Q^6q72bw&L3{aC|2I4Fo*?NWN$ zVaEU~Zs*I?BXn)hs|PMQb2#Jw}Qb?>py-00PwO#t;u(mza9V^ zazXtaldY4Of32)N8NjO-Y@c%eb+@A0u>db;aDbb-{^eN!+VU%wFW~=pyW{*afqsP+ zY|i$-ro$SbRD+2S(?8xNKZrG1;Qft>o@KnhJZwOzmh8MD|M~;AOn^=Vn-7y`e|ai^ zQUUeb*nj*%grFS2%GaWkEB^_A#O#gbGzdV*esS8$aA%B}U z0j&HQeAfEk+c^KPOC})Jh-BZ8KtSH#^XKFPKtOn;e&4b>^@egvUQE9IEP{ZnO(+OG z?RnlAsRO@+^$9g~37YZlqYRIaClwU*D^<3|q9iH}EYjQjlcWdjh5#lm+fhVSN&J`LfJCpq|9bU@#UK9v zD^rpJgqN3B(0g57-Hq;}Xs@S*b$`F-%8Z4fTS!|BZYNGZvu~{G3VKz?hiT~6zU~HRV$28K`amW zfRHpd_c1&+GI6!pMjA)Z6{LryEEO?UKwI4d&o3h>qw5xNo#6X&rN)dP zLuMHng+#c6Nl6RoQIXKP$*h~cV^PVzY#vnCR~&BjBaxxKd(d-{3JZz3t?iFpD|S-Nl{;Zuna9WF9cXwT# z^S;OWg!G{x*^Y9xa{778xuj1en6)lLSDR)#5YKO&Ui zZof`*18>AUiCVj|$&UI&rjC&@1}_3wn{(4I-~UvPbQxvi$Ozebm+b%w~N0JA4_ z@-&qr+&elF_HrtRDE9AzrSuP}( zDDBnui}L6-;>;9@NYw0Oz$V2ajR=yH7q*Y4ArY(wgDiy!eiFsR3Z@cXNM^~rA!r_Cv#5sk-j4^o*b| zZtg)L+=0>kops-06nx!O)|d7;WT&$Y<+l4XiqbS*Q}4H0`N`IF%H#D4F&gZTEK0|< zE_)8hNvls*kdH`G!q+L4#t%y{4p05|8jW7zYpu>S&vVp-ADHro4u7^<^=h?3qY9ZP zWydGJM802y$X{VUjB7J@!Z5dZU3z)%pG0m)J8oPs?UGWr)viUwrrPqqQ$lUq#}t2D zEi|=BR2ESY+n zr#QFFY)zb>R**}1LiqM`_-kMta2gz_iFqJEmk_csZiPnUBlY2)98o~sNfWOOZ4yku z?)HFc3zU6zI$qb6nt7V1`3b+`e>&~~V1j`6mmpCrbud0$sjt4l=LE>(h9^%PtCypu zM?hi>X<%U_Oh;!L_hye$RQ_Vp*XW!oTRe=Fn|z>4y^jBGE3DIl1FC zbe-k+*vepqQeApK#+Kq)~xZ%CFSb|Ou9ugh53V5k{vZ>w3T%?yL zq5`T6G5!7U$v_ac)|}o05xaX3>akslgwIRKVltipbdUu8YE>x$dKyO`S$qxp!t zpV!}|ajkfdCy&`|H)J>5c8gIzex&q$UiFl=;@nJDAqv57j)mzh4IMFBDzQh!o=2v6 z=SdTsqH;t8PppDF%u3u(CFm^9D|%726fusPLPpxP;|*+#x>zvu#Y3SN0?}ESt$_tWH83fJynX#qBKXLH$BqO*dc7 zx;5(9>w0;JYtkM+KzgJECb8giV$n*k6s(IMlOy_;4(E2G&7zLDf}5rR(rM~mB!*sH ztfRos_;W5nN^f~D7aTiIN}2tsUW;AvJ1!NOLAe~_`SQgiE|#vidONO~?a?5U2=JC`T2+^G(j<+MO&)ho)4Fp zr8DkvsdjC6Ki?~+RUH6=Wz*k?SBh(KmzpIoym`i2RkwBh6`U*PB&8+u)abrcoZ*B5 zn*0O$ns}g-gUV1;v))7`B%P7dTXlV_rjBvHJY=q7hX_6j<%wmw004Z2RX=R`pBY2( zMxz79Y=WsY@-mK>EKtE!n`)u=`uuYp^@YT;ebq~lDOJlHkL^fhaeG<^SGQOEY}|l1E-}vhjO9I>B>yC zE&;ZHPnK4@i`?6R??wVp-8vMeVlWU^QfhZ;dD-Xa4w0)QCGfr}rTub;m$NCh0(->^ z$rW;YOKBk7B$dMc_(*UJ;k~SEiOGQyUIQPYk_I*8~5-M*WoCD@!(TjIvm$ZEiE#de1*)uFWkT$;I7_S7;=Yj zl&{qR5{<>90=16^>!Sa3DPH92bw;e@c;e*xaX&N+28y$KsJC|}-`1Bbj?^T#^FGEW z1=Mh>YM+BN`DWc^j=^Wn%jLs5y*qeS>ti;vlDmi9?zhiHT{jtJmBq0H$~%Hz8c^9! z7%^BsV-0zKK>We=-6DT@*Lgu`TN_u-MBV#N*CA1M7NUl`1spBg7-@V^O=J1m)G<2aySa6 zOZ|KA&$;tlQ@dd>8rCXKB;)~M+tW1qAuc?fwzm{J=x|ML1}T=NZ3r8 zn$72V{Q=n_wYZ2Rq={#9ouN11%W^J}2)eFan!t9D(W*iY4VF86qIa|i@VtILg!i3y zzdpfnA+ZG~Qbbx_K#Um86?5-Uuy6YScWyiqWw*339cxb9c+5*)rV z#g&RE)qc3@&adUc>x2%c+|Kb4SS9seQMEWd$v>&<9E65Roq5(GNXla=-@N6%Gu$se z-R}~U&m&{9AfK@{%}0n`>w%y(+9dgz`<3i4VT6e$meXUj4Qe(XgXFUV_{9Ybw3240 zZa|)6e@_8Hzr+_s%DoBA#uNsDhffBO42J=fLqhlOdDexV-YYcK90K-6cA>S>c^vAmA3o&%5*H5GXVfI zh?P}M6Ix(#9nuIm)KvknuKpZFm+By}JJK6~N!Ostr;rsNpJI>3{wkBN-t8|<`G_5; znyr-4Uw~s^iZPW*r((V4J@E=P^b6K_^DKM7VUN{fMN3=JQ?dAan2-Zl!ZJbrrFL_U zH|sCg?`7z1n)=!1oLJy2uk*P?MZZWO`f^6!CVTsJ5I{w?K*8X>2yW+Q702>_^0k)M)J|pU@?UDgB+|>zs`0WY=0y#@BRWtX2dgS9@1v>VK7Qq zSkN#U=4W5PS*Bk9=s?eWKO}VZcSzcB7;TBj?(A^Fm&7CaD&^*!s2onQgBS(@H+Aa| z^-(Q7T+T0mT0>bg&!dG(8Uo)J{;)w%=UJ^z4>}InCl13+a``O6?uUU-S9hzv@dJ^m z(nB$LKXu>sQXTG2xX6h4JY7<|ygkRBZa>*iObKpi(bw=Y0LGE{V$$rc&RO)J14N(8 zzSWxSG2bquK@H-IvPY>25O`BcAm@1?pbVVQ(n1oE-=ef%%_SYEu0Q;;^_`wP#Q!;R z+%i1I66R=L@^kMY{yX(5Pd_PYE*juN?UM*w-l!wWiO2V&^XWtFH=a;hCbo+6JR+P2 z>5vYGLjxla)g*%ME9L0890ALa!SahPS~4-&obV2&1{_T-KSVpqo1>z& zEm9s(Q`ujKe&_}K7;5(%S};IC8$5M7oBp!U+vutW6GDR=)jJBGcLoR6f2YmtnsSFw zJn4imWA?d<>%_t28Q=88L-|CifV-al(SpbaB8SJVEHCw2Onu2PH(I5XXY+0A#eHZRUr>T4WafSH(#o1Sy%Z(zLnCO^}~^ zTyom4Aa6r5aR;TuoIo?49{(5Guy}%xXdld*%r~Wyv&7QUuevk8kvV_gGmdfakh73ATuSvO z+!#)el=0yZ%0*iDAtXIOHn^O#x7u1zw(X~o?2=C4HjdE``gy7k4Xv0Y8WXjBln$&ayP1l54~B99*Np3SlYb zCAEJOUqbd*)mK}anK^SD-jR`?xL!WA$=~~&^0N`M+oeP4*O8|#ThGw9TMS@@W!<8& zjMN%Le{$^du~+M6;xP_**8G3%on>59-xl|Ykw%B^Zpi`Z5{7nYkZw>AL`u3F=^l`j zMpC-FJCyE{E~Vo+-h2Po`|^4He4aNmpIM(XXYX12?6db--``pj&;8PhOJDxL?)c&K zQi!I?QK&oc6UO)0_hd8HC#)&^t7|z<4)aJv{W74J4i>PQz;uuGsxi8Bc1((MXTDcE zQVNgETiwbJ4^fSdv)aHsWE2&~mME;0D@{)p9oH6Dn#`d|eBEP628^0HNSR%RZ_Fy! zpz5Sy(l4m#tn&)}`I9a%^KfFYOfemC0&=bH@4aH$!=)O!g_@rI1V)HS%mV81ojlzx z#goX(G?4!fgy&?J_$*H$_WFX0v%6rqqK(1q5@B@PD&0`;w^_e9a$gDup+wj=?Up}i zyz#!k(f8jTw6Cbtf>LvWrXOhK5=h?DBV$>Pd}snu4n5g(DjdUNGxE)Hnx~oH18x@? zL8g+OQ2yo|!$v&3=ah?#{z3L;POOlO2z!g+Mte>eN*3u*&#$geVLx{^GTmCIL&Js= z@et!q5xL(#EoKezNZizfD@tjNj`=+BTu)^_u)RtePn=R+_B_u7zwl67cH1$?C7(%3 zHguN&BvveK_lJ+ZCC03Qvdmm=O9fe(ZsZUXCKIT55K3Nj)Dg1Mue51F2)977M`r6J zvg*RhmjL@tdnU%=G~;5{Dup!3utUQ=(ZqQ<&qCW6SFW%AR$n!cU9p!xBPw7uZc0^o z^zoJWwoTz{8xGMhrY2T7*Bp$OkW-=e7N=myZRC$wbfs#3`?w)=_{-@dvxx)(Jt~sA zj=YNYU2ILqEPj3lM;Z^N!XS+oii@cO(r;tFzQfzwm?|L{OZli83=B!kIPy$3>=?n0D7%vY*%9R*lUex)4~ zBNq9#A!Ec-*|WAG-rx&6|6rX`Zh|}AmSBYno0hGZ4+b6ODf3K*4|ANFM;7lcAi0*p zLHBvQH@eE*i=JovnBE`lfPr0AMYCJ$6D=Z@;!o0iMKlyL%0lwVlR%9+8+x}>2CLp@q@mX^-$ z5F)E542=4#Dkh@g0tp1O82(n>)sfYQLF~oEGy%W6o~_8bLJ`pVjLgrsN9vcmw2HOL z%bFi2U-VDuyK=Ws67n9LO`Nox$#@4+&8xR<^f% zT(;pGvA!S^!D@1Go-VXGLK_soh|A~?#@6)DAv2?yPQ9upn^W+E!f0Q@mC^;u@c2;M!vMrrBwFKVW@xco|&3HVZ5y(q5hxKZr8@5z?J0ruh=Fm7I^d z17qju-y}@eS3VPdx?w;(ood8_ZrZ7Y7yj14v$UD0PU~$1sB%-ZFzFW+Zcgs*Q1nBY zz?ncT1#)Tq3Z{XfUY1;}_F3vvh)K&_#^U0Le1pX&?)&;9mR^f6IF~`a&JZ7(`)BI# znC+2j$sK@tHd%z|?@)^&GVoX7nQjyvByRHk=^_C=!U8B0Q|M%?;1mL@uM z_X&Frn0MhP4e=~Y1EnXseM|MR=|K{vk~O}kctopd}uTe3FP#zct*> zw$xcr`1Of?=}I$6fkcFGo$UlZ2Va>wxv3<|NLZgq<8b;8-$BB4ip$v8F6$vyGT23D zsml6uiB2n9U+$2#16O3blC;$UsfQv@;~9)NIdT%saq|)l)O5^2$_XDZgfAUo!SwQ6dbT zi2FVmE55AdrsIvd<=uL5Ubx2A7*Pb()fa_CE=_1}1W6u7Uy06&X+Vv7V+W~ApvBE` ze=Nk|;qFPj<+7x&bqNRl`MhzYBOSrI-3_34To-@5Xvh4fLo8rY4j*a0hzAgdJk4B5{xHau87;A zg#v{*`^K|uvAWbDmCIfXurH-k$vGT#pofh0tT@>*S{h~s=HNPma+~~FI*!H$9p|!l z2TeqGx7V`P-R_#X{bL+v0xnGuo+l>C5-cZk)2n}UYtCUlCEwoC>W=buVt^#CsIpyA z@ITo|+An!~)VTpaSB}uXO%7zS^(4G;nVVR|4Dh-p&;3ZRpH^>W6UyVFGHeivt@ZM$ z_gzNc1Xsm!Ge8xzA?rt0r&$GUh#%j^(^Sg+kGRB8dvO|XNAH2)?zGep_5vdpkzOS{ky~^mc`zhWMM&v{JGXd)plL!Ekte#)4M;=YvLa!W(K}VN$m%#`) zS&;1EVF3n|+T|xKCZxtf{;O9M*u(^hpZPx&o;f4hiZb1-fL{P1>n}8`;6yB#C0z5o z87+EEeDoQEsO(9w#_6saXV`x7uPO(~zzBz4%{vxV)mSv+t>`P04l*LnWKN@4uiFc{ zCs&OedlV<6d=ZmyJnvQ$WN*rzk?+Gh;SmwQoUrE9uP2^MM`sBBIs?W-g?aFT29c07 z8s7!a;Mj_fAQ+czGxMCg@Yg3}^Vk(Or`+bVRVRrC(5SS6sZ*#M`b#I13GgG}35iJ+ zJ6s?vL})ws&)Y|Iw;(t5T|H5>C@Sv+0 z0RR5~LH{={pj3u=dU|y;Av`|6`c&Q)9akth{mXqR43 z;}TOCQdAVC2cZEX2V!w}i@dM0!{^xt=issL* zKsYCirjo?+O58_p1DZQ8k&;B}Y!n1E_!PM22bJ_@ z;}b+d&9QKw4&1sy3i8^?T%xC(DW z&0!J|9-caEa3{1z*W<|_-u{K86gMx>w+McUIAt3-t#NlQIbWk@Lpz!)9b}>; z?Yw`mVYNp`aJc*QE07AO_qy5(hOxMG_!63B15E#1o9X~j={c$Pg;4nmF~-W-KKg<1 zOUT4gce>I3^2#-V`0`t;E(^z)Idu}G1{^NaI5M2u1+c5HFFy|6(ltR_V!RWd{E@-w zS-Md=U6?-aySrlT@_!_Q*pCB`AXzpvV8T64gVfg? zXHbfk82X*^0{nmGNOgD_J#QD%nO8S^W!;u7hAHo$X||gGy~4N2tR&wwuAenoZISU! zLX>qKx*Kye;o<`qxj z6wH%hNk2+*T-wC18i~~&mzx(Ot^@u3{g*cvJC!`EgV30i2HTOOV|wF744{*cKghmsBxYaTjXvZuiW&0t#ZaTE@L!bn;bfsT#4{^gSvC1dz0=b_+{0mQF-2cx^YLvC5N`PS z)1I1VXY-Z;)8qvATzs~5fCy2ET=VNp%#8Q-2ZxP40t$`MjsbZWi@u~thl^?Xog9Zp zg(q3I_bA*pzg)v2qLPjYhZ7AJ_+B~0DKfqo{Y^_Iw<6I+Uu`PgKtjYsvb(F+@|yi) zlWhl*>EluQufv1|@9tAV5(4%X)Tc?~;|jP`K2WC0y_!yfx!4{9yb7IJ@%_=JRSb!gbVNX=G5Kh!E6Eaob$pkeG9d!{1TRKlvb zM~iXCgrn-m?eU}{!1(_(f9eY<2NT1Ql?Rg(yNdfatvLI`%w)|+TwWnOq?7k;K6FXH z@zfNGww+`%GVLTlo4IfBauCU6;GB`Va4AFnB|wTZld1fpuHc!f!efz2w!akbJ6a&p zbVTC-N12Ox9M)Vz%Jd36>S_%LR+u7W4|=ycPw>j(@}g7*hMic(dCl+TteKsI7R z^U}k|{Z*d<3$|Nyy$xDI+^DF#eADO>K2@X?@Cl6BX2YdQr& zyX1Zuu(_b~Iv|03u}N5&I(@^<8z^tYJt+l-yJLSb?=P%;#=>|_0%H9}K2;UAlTm`=yXott*qVta)z1QQ4jYx{LZcP>yj(}GPk5h?VUlVO!O>}+x zuybLv0sCz}eljshUc4*Z_V_{Gzfps7fa5*hQht2KV5>*C`kYoL@VHgG z=I#AZ@9o=LoC_UcgPkgvu0Qo1@=`I#zuzoyIQk{ez{tO`vG{D@wDtL{Ws{OY9w0{$ z(&iaQsI&9*f`OT`YsV9znUr4hEjQ0Pi)%lP&!nfuVeGnw+%V3qwxD3e{p?k=m{?2! zbYYEy!_7GI0~L^sk2T-W@vxb)yrHJKYT9ry)UBUh^`mR|9#xU=qfmdXiWlmtBBDhX zvhKGTagZ7(X5@2Fpy)R6*gq?KQ%RN1x|HgCWcEg#0uGjVX6^G0t~ z{C0t|E^uKoMu6Pe@vrnf-gkot;Kuo%W*RW=Ns{)f3RR9lQPIP+J1&fX5mI6vg(XBE zFU(Esc(o(>o~b?E>7BgNGQ+B*=X=$%5xefYZy)LRUwdmYhM}Xo07|MnZw=`CDzcd} z45Bgp3UYB=%qNve^Vr2=Fs>5`w5~189gBtDj}C`}7McWMn#9xEqxKYyPBWEVf|0+= z)yp#|>#%Ii43m$(C@s08b&*&m%h}oLH_?8rXfL>J@!Ba9GU7<#mn}1Fj45r!(Yk<) z7690e?bMzhHTtRf`fMJxxN! zKtGB8M|{5wU>YU9x*DV_|$_kn*Wh$(>iOjBEYfW+cDG z(xpW_T&3kLaHA(eX`Kv8fy1c*kewtPicJEs#nre2-+7@ZXg6yVYv_>jTcajLg&E=qp<y6?!~*EPkXju9le$^M8nK`a@vuyMC~+$_EJ zMCZM;5B6K#=u0|+IG{M}o3xO~4C*=>Eshl7SAIj7)P}5IymKScf3cCCeqswcj*5sa z3I&8As)-x)UuKKo<5WcW__P}8xD((PUI2cJiQQnbs=iIawx*8Oj0GZxKk!>V09b5} zIj6Ph*Fh;;Ja%Sk9BmGh7<+*-`JXkXFjrx2jtXaH7ap;=A2-93+%h)H$8q`Ec>y7P zdJ{igUmkZ?DaRC;f~g$l*0NH*S2Zy}888f;OM(j951jKpG*y$-LP2WGQ{#diPl<)d zStCj3Q5)v7KbMQ(;wkhv#ZQ**;WrAVByNhOa1+PuS;QxmCd_`29$(M)@&JSN>NQgx zpDDZNd0}7;2I1KBiROZu%({gtSM0H#QVC0Z&;LG@Ozp7?=O_^=8V&M#J&Jq)b$S#G zG8gBUsn+4Op5d744jiK*SkgA&NGl-MdTKUaXgI#rWI4*;Z!1DSP{Cs3_@?;cr?AQZ z=zDl7wUHU4qUz|wp;o4eiR&v9le?9E^mwO-jXNhIbN`uAGzYxsAo@NEqGT8-J6)Ii zLv3*)j-z5>>*t zC`utX-KSI*P1yM%)~@w(1n1^(5uRHyLoU{l z$urx~35{Ln8c}rdBY`5lZ3qn*LgoVng;)Pv$<7EP%*?0@} zco=4(_#6ebL2We}##^_M4&q5WcWm@DIbE0hkNWae~2TeRFy z8yGqd^~%uuu0_CR?~RAitR|cD4i>-3*GevP7HO#OP+FWO-UmE~3d0ta8l2YBq6!xH zKL}CnO3tWD@ViJ^J1HfQziY9ZyD&&*SByRqB5J*P-^ok~|keQ0u;9{$yL zV7w5&oV7E*Q#xQP0Y%;fRoP9wxmI+yd67@4mcPE)FXB_M z$;jwb86*9p=Qmjg__aS(6h#2rr2VDJr0KhZXfWB!BAef5dG1&N6WgBGML_8)4f#mx z=!9&wJY8PRx0V`wr3R&-9l&cD8nb-nvj($#ms(6IC-NDBZu%P@iwnI)L>rF9GldLO zhHeI7a}K#hZdJmvkj{3C$2{@RL0)ljOt%_F2XyT>eYCxytoS!hVGOs7m)pj#=NNKJ zrx*Gn7oJ_VgR)_LC2x zI0hpc!Rfd@8^NqHBO?T`g>GprRy<^3M@2Di_UyX@ z2{wx3Q}|k27bT`GW{*4I@xB$3QQ4+}kV_7l2>~ps*&GbAET+w}9~46GE|AS$rp0}R z#{_cO27SWbKwx&O?C9@r5FfmJMOd`YjL&Kose|aYwrvtYfhLRkrNb$Y^=;AD5bTlT zJsFwZ)$C&zF>TI(Sdr3-_i{?#3|_339CGn ztZ$9&z!#7(waMeM3_G8AO0OFU77CQ_yN(~@-K~?zAgU3*&+HvAh!9G6cpkKwtJFu| zR?uBKXA8;RLX(}`m^mWqY37i%o(3E?oH%mPKFxr^-c6U}9Z?ilV!rjkp1DF!{GuC4 zYBwRmZUWC)_PD*Aq#U3yz**?G5r1y2G@uCk8Mc~Xtie`JZObio)Of*P9Ji*zL^OJ% zizjKB*?75IZhI4O>uk+Od>ObVB1;6`+3X|)n2@-Jd|9V;078Vvwjk(>Vcb^ujyw_) z8ejvw=|9{G4HV0zXcqI2 z3x;OwU__S0LfD(%fcW}T(R79g@Hfk67+-C5clouy-J)KjmwNN64lnOp+v8~|dHTNU z*b&Q$z5m*$A7QBLjyz(%Ul$&rrsibCcZ~I{gJ$kbm05b>R~2Io$>92%jO&RZH288c z9ww71N^9awC-oUhzAe_(+ymHMz4{KV<+&AI za6?2x=24MXX#lh=(6#L~NtX+vOti}_(>KhT^8}Yf{zCa*49xa=Bdr@#%rpzH$jz`g zSC@^_rsO_Q-$J?~kj1{7n1w#;v=5*QomwrW6N#%G~cx@jE2FZ{-PxdVD`~kJ-r6H!^=& z1KA{zBqpCnBu<`VE>yhxy(h#7Uz2ADWNFl&?rwK{DeVJPMojI2_V~Q;HzM(`aC#t5 z@4C`S&T#tual9hb0<-md&wWg+nUk4VZ=F44M7ylxDq@BXkrrZNk|o4H3gmQDap1Pz zofsHs$LHebxl%xAJj{kWfd_+6zEgCh=DI4$Xk&S%6Ad1mnw(d{xcccC&~}QcLOJ|G z_zQ%p%?}tA6C?);8}Nvc@AdN*y7&P`6sDarYpu*!d_t(fjdr703=DT^b1VU-GJfFi zaiBZzC&+VGSXhA;jX6zJ>3tp!CKm|m#Uv5Ke$sU9e$twI7!eynSgEWkpPO}@Ip5uq zK-`E0qwo~QoDyY>G3}Mu*j|jkt<>EV!~a#4JcFKGM$wqnr|A7!6Yl|mzAs6baxuhT zubfKIwf*ZxBju6kGcg*LHE#iEWhA0Ok@tf)DXZKQ4uabafQiNGsF1+dG+%o+Ct~W( ztLEQ1L{TFG8*gmBNnz&FRp&MG-uoK4!k=%}$Yl~UIE-bBJ}kAyKR^FoJfra>@{<^{ z4hhkKcPIO9ikPVJEZRpzZf&?!lB<2kG9V@eHH{83GD;=bQrV(*2vnqG`slOqj_w1S zB{LZ`VUl~BJKoi1e-0|Ak+fstkHgedd|+W(bQB+9UtGeDFV>+Te{t9DYSx>k<1E$V zRRM*QX7MG#Bfqc65@N(zy=aypCTC>_FTR6~Bx(oYB+m3lyB(S^3^kjU-5K^U%t(g) zLF`C@`cS4YeUaR*&-OM2X~8^ihZ0uWX3GTzVu&59+RpY;?T!%-l=hIT?BjZCSDrB-r5P;fAP=y9GCq;^)*tm z`sZ#02j}9~s7nZM*)l%9xrU-dPxlwXIfI2u8E)$-eysyrw3?<@7|Ug6)cAIB$X zG}=3ft*zl)wxkMXM#tL&>35^gT7Gh(cyRYMQ7sCX$KE=>_bEhnQ~g-_{69hEEiA<6 z0*h(y!F(<`O_t~Lnmrji0Ms*OT1WZdv<8Cm3i=%$oU6Z502bQGe*)1ji)h+6pS^iF z(TY~4sa*Rpa_19Xmi5e}J2y*?U`?RWotWxnN~GYL1%qobnr5Z?KXA}51e_(@Kez7d zNWRm!PDsbtBRxLK|6g|KhYJOGLzsjW5rCQg!xR11|A2*hwE63Q8V9lnfhaJ91;TtN z`9C&&fWY7rNeHj}1&<0?K>VHoCC{M$ukBMfz(m$*EqBQNF z`D?QWh=ZCw=+N^&SQB_-h5$_@5GOd({Ev+T01ThPv(i8OIn%#S$nXcd;08=`nE$o) z{2?j!IMwHX_V?eM6q?ly&@DAnmHD85ek&q0tNza~?oG^hHXhH!?p(*CC z4WG!5NdX4r!2l(*K_5?8Y()?Lr=urGAZ{KWdX`<=q95H%A|kY`<-$PyZ3ScF3LqvxUUo4-BVLh~yrVUt%@j-Q@RP%Hh|+TGpPCwq5~Oi!KVe|4kA3)#xm(Chr0rkS7$ zl2a&5%FWN;v$hg@103JL0t8n+WoB}UaXYN9nwJ09v2HJYbLr{m3??U!)_8NKzupCO z2M(f#pX=@^oAdu=#(vyE2m|O3(Q$8o$%cvTKf{g3ZTW04{CCiRH x@QN5VAKLy;8)$!QNaVMJe>+3TiS$V2m6n(Bp-eOl_!|QRd1)o7GD&0q{{u&r!_WW# literal 0 HcmV?d00001 diff --git a/docs/developer/architecture/models/alert-history.dsl b/docs/developer/architecture/models/alert-history.dsl new file mode 100644 index 0000000000..c399229435 --- /dev/null +++ b/docs/developer/architecture/models/alert-history.dsl @@ -0,0 +1,39 @@ +workspace "MetalK8s Alert History" "This is a model of the alert history system." { + model { + user = person "User" "A MetalK8s user." + group "MetalK8s Cluster" { + logging = softwareSystem "MetalK8s Logging" { + fluentbit = container "Fluent Bit" "Service collecting and enriching logs." + loki = container "Loki" "Service indexing and storing logs." + } + monitoring = softwareSystem "MetalK8s Monitoring" { + logger = container "Alert Logger" "Service logging alerts received from Alertmanager." + alertmanager = container "Alertmanager" "Service deduplicating, grouping and forwarding alerts." + prometheus = container "Prometheus" "Service monitoring the cluster components." + grafana = container "Grafana" "Service displaying metrics and logs through dashboards." + } + } + + prometheus -> alertmanager "Sends alerts to" + alertmanager -> logger "Forwards alerts to" + fluentbit -> logger "Reads logs from" + fluentbit -> loki "Forwards logs to" + user -> grafana "Consults alerts dashboard" + grafana -> prometheus "Queries metrics from" + grafana -> loki "Queries logs (alerts) from" + } + + views { + systemContext monitoring "SystemContext" "An overview of the alert history system." { + title "Alert history - System Context" + include * + autoLayout + } + + container monitoring "Container" "A detailed view of the alert history system." { + title "Alert history - Containers" + include * + } + + } +} \ No newline at end of file From 217bd5ab00e978120816a1b1a19af4db773bae5a Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Fri, 19 Mar 2021 12:08:18 +0100 Subject: [PATCH 03/17] docs: Add alert-history to architecture doc index Also sort the document by ascii order --- docs/developer/architecture/index.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/developer/architecture/index.rst b/docs/developer/architecture/index.rst index ffd5e0e77f..aeb173f661 100644 --- a/docs/developer/architecture/index.rst +++ b/docs/developer/architecture/index.rst @@ -3,15 +3,16 @@ Architecture Documents .. toctree:: - authentication - deployment - monitoring - configurations + alert-history alerting + authentication centralized-cli + ci + configurations + deployment + logs metalk8s-ui + monitoring requirements solutions - ci volume - logs From 0db7354310ac86e24ddd4a0531580e7a371c528e Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Mon, 8 Mar 2021 11:18:08 +0100 Subject: [PATCH 04/17] images: Add metalk8s-alert-logger This is a simple HTTP server, listening on a port (default to 19094), waiting for HTTP post request from alertmanager. It then logs the content of these requests to stdout. Refs: #3180 --- images/metalk8s-alert-logger/go.mod | 5 +++ images/metalk8s-alert-logger/main.go | 50 ++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100644 images/metalk8s-alert-logger/go.mod create mode 100644 images/metalk8s-alert-logger/main.go diff --git a/images/metalk8s-alert-logger/go.mod b/images/metalk8s-alert-logger/go.mod new file mode 100644 index 0000000000..bccee28c4b --- /dev/null +++ b/images/metalk8s-alert-logger/go.mod @@ -0,0 +1,5 @@ +module metalk8s-alert-logger + +go 1.16 + +require github.com/prometheus/alertmanager @@ALERTMANAGER_VERSION@@ diff --git a/images/metalk8s-alert-logger/main.go b/images/metalk8s-alert-logger/main.go new file mode 100644 index 0000000000..6d4f41f679 --- /dev/null +++ b/images/metalk8s-alert-logger/main.go @@ -0,0 +1,50 @@ +package main + +import ( + "encoding/json" + "flag" + "log" + "net/http" + "os" + + "github.com/prometheus/alertmanager/template" +) + +func main() { + address := flag.String("address", ":19094", "address and port of service") + flag.Parse() + + log.SetFlags(log.Flags() &^ (log.Ldate | log.Ltime)) + + http.HandleFunc("/", logAlert) + http.HandleFunc("/ready", serverIsRunning) + http.HandleFunc("/health", serverIsRunning) + if err := http.ListenAndServe(*address, nil); err != nil { + log.Fatalf("Failed to start HTTP server: %v", err) + } +} + +func serverIsRunning(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNoContent) +} + +func logAlert(w http.ResponseWriter, r *http.Request) { + var alerts template.Data + + if err := json.NewDecoder(r.Body).Decode(&alerts); err != nil { + log.Printf("Unable to parse HTTP body: %s", err) + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + for _, alert := range alerts.Alerts { + encoded_alert, err := json.Marshal(alert) + if err != nil { + log.Println(os.Stderr, "Invalid alert format: %v", err) + } else { + log.Println(string(encoded_alert)) + } + } + + w.WriteHeader(http.StatusNoContent) +} From 91f3eb32938b64664f5c0cec786bb3dab75791a4 Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Fri, 5 Mar 2021 11:47:23 +0100 Subject: [PATCH 05/17] images: Add metalk8s-alert-logger container image This container image is used for alert history. Refs: #3180 --- images/metalk8s-alert-logger/Dockerfile | 30 +++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 images/metalk8s-alert-logger/Dockerfile diff --git a/images/metalk8s-alert-logger/Dockerfile b/images/metalk8s-alert-logger/Dockerfile new file mode 100644 index 0000000000..823de2d3c3 --- /dev/null +++ b/images/metalk8s-alert-logger/Dockerfile @@ -0,0 +1,30 @@ +ARG BUILD_IMAGE_NAME=golang +ARG BUILD_IMAGE_TAG=1.16.0-alpine +ARG RUN_IMAGE_NAME=alpine +ARG RUN_IMAGE_TAG=3.13.2 +FROM ${BUILD_IMAGE_NAME}:${BUILD_IMAGE_TAG} AS builder + +ENV CGO_ENABLED=0 + +ARG ALERTMANAGER_VERSION=latest +ARG PKG_PATH=/go/src/metalk8s-alert-logger/ + +RUN mkdir -p "$PKG_PATH" + +COPY main.go go.mod "$PKG_PATH" + +WORKDIR "$PKG_PATH" + +RUN sed -i "s/@@ALERTMANAGER_VERSION@@/$ALERTMANAGER_VERSION/g" go.mod \ + && go mod tidy \ + && go install + +FROM ${RUN_IMAGE_NAME}:${RUN_IMAGE_TAG} + +MAINTAINER moonshot-platform + +COPY --from=builder /go/bin/ /usr/bin/ + +EXPOSE 19094 + +ENTRYPOINT ["metalk8s-alert-logger"] From acc2e536d79185f6bdbcc2aafa3b317884b678ad Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Mon, 8 Mar 2021 11:21:12 +0100 Subject: [PATCH 06/17] build: Add build of metalk8s-alert-logger container image Refs: #3180 --- buildchain/buildchain/image.py | 3 +++ buildchain/buildchain/versions.py | 5 +++++ 2 files changed, 8 insertions(+) diff --git a/buildchain/buildchain/image.py b/buildchain/buildchain/image.py index 182ae6a8d8..1c69bea945 100644 --- a/buildchain/buildchain/image.py +++ b/buildchain/buildchain/image.py @@ -236,6 +236,9 @@ def _operator_image(name: str, **kwargs: Any) -> targets.OperatorImage: # }}} # Container images to build {{{ TO_BUILD: Tuple[targets.LocalImage, ...] = ( + _local_image( + name="metalk8s-alert-logger", + ), _local_image( name="salt-master", build_args={"SALT_VERSION": versions.SALT_VERSION}, diff --git a/buildchain/buildchain/versions.py b/buildchain/buildchain/versions.py index 6c2007797f..2bf17325a9 100644 --- a/buildchain/buildchain/versions.py +++ b/buildchain/buildchain/versions.py @@ -205,6 +205,11 @@ def _version_prefix(version: str, prefix: str = "v") -> str: digest="sha256:240b10b07e15e95c3009da938e3abb8bef2fa47ea1f719ae58f7dd116bcb2f10", ), # Local images + Image( + name="metalk8s-alert-logger", + version=VERSION, + digest=None, + ), Image( name="metalk8s-ui", version=VERSION, From 4a23e507cf4cf4f13af15adc073e69edde276c0e Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Mon, 8 Mar 2021 15:49:58 +0100 Subject: [PATCH 07/17] salt: Add metalk8s-alert-logger deployment Refs: #3180 --- buildchain/buildchain/salt_tree.py | 3 + .../alert-logger/deployed/deployment.sls | 58 +++++++++++++++++++ .../addons/alert-logger/deployed/init.sls | 4 ++ .../addons/alert-logger/deployed/service.sls | 21 +++++++ salt/metalk8s/deployed.sls | 1 + 5 files changed, 87 insertions(+) create mode 100644 salt/metalk8s/addons/alert-logger/deployed/deployment.sls create mode 100644 salt/metalk8s/addons/alert-logger/deployed/init.sls create mode 100644 salt/metalk8s/addons/alert-logger/deployed/service.sls diff --git a/buildchain/buildchain/salt_tree.py b/buildchain/buildchain/salt_tree.py index 8cc496bf88..8a5859db4f 100644 --- a/buildchain/buildchain/salt_tree.py +++ b/buildchain/buildchain/salt_tree.py @@ -206,6 +206,9 @@ def _get_parts(self) -> Iterator[str]: data=versions.SALT_VERSIONS_JSON, renderer=targets.Renderer.JSON, ), + Path("salt/metalk8s/addons/alert-logger/deployed/deployment.sls"), + Path("salt/metalk8s/addons/alert-logger/deployed/init.sls"), + Path("salt/metalk8s/addons/alert-logger/deployed/service.sls"), Path("salt/metalk8s/addons/dex/ca/init.sls"), Path("salt/metalk8s/addons/dex/ca/installed.sls"), Path("salt/metalk8s/addons/dex/ca/advertised.sls"), diff --git a/salt/metalk8s/addons/alert-logger/deployed/deployment.sls b/salt/metalk8s/addons/alert-logger/deployed/deployment.sls new file mode 100644 index 0000000000..5a7e1f0dc1 --- /dev/null +++ b/salt/metalk8s/addons/alert-logger/deployed/deployment.sls @@ -0,0 +1,58 @@ +#!jinja | metalk8s_kubernetes + +{%- from "metalk8s/repo/macro.sls" import build_image_name with context %} + +{%- raw %} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: metalk8s-alert-logger + namespace: metalk8s-monitoring + labels: + app: metalk8s-alert-logger + heritage: metalk8s + app.kubernetes.io/name: metalk8s-alert-logger + app.kubernetes.io/part-of: metalk8s + app.kubernetes.io/managed-by: salt +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: metalk8s-alert-logger + template: + metadata: + labels: + app: metalk8s-alert-logger + heritage: metalk8s + app.kubernetes.io/name: metalk8s-alert-logger + app.kubernetes.io/part-of: metalk8s + app.kubernetes.io/managed-by: salt + spec: + tolerations: + - key: "node-role.kubernetes.io/bootstrap" + operator: "Exists" + effect: "NoSchedule" + - key: "node-role.kubernetes.io/infra" + operator: "Exists" + effect: "NoSchedule" + nodeSelector: + node-role.kubernetes.io/infra: '' + containers: + - name: metalk8s-alert-logger + image: {% endraw %}{{ build_image_name('metalk8s-alert-logger') }}{% raw %} + imagePullPolicy: IfNotPresent + ports: + - containerPort: 19094 + name: http + protocol: TCP + livenessProbe: + httpGet: + path: /health + port: http + scheme: HTTP + readinessProbe: + httpGet: + path: /ready + port: http + scheme: HTTP +{%- endraw %} diff --git a/salt/metalk8s/addons/alert-logger/deployed/init.sls b/salt/metalk8s/addons/alert-logger/deployed/init.sls new file mode 100644 index 0000000000..4aed65398e --- /dev/null +++ b/salt/metalk8s/addons/alert-logger/deployed/init.sls @@ -0,0 +1,4 @@ +include: +- metalk8s.addons.prometheus-operator.deployed.namespace +- .service +- .deployment diff --git a/salt/metalk8s/addons/alert-logger/deployed/service.sls b/salt/metalk8s/addons/alert-logger/deployed/service.sls new file mode 100644 index 0000000000..dc785d7d0a --- /dev/null +++ b/salt/metalk8s/addons/alert-logger/deployed/service.sls @@ -0,0 +1,21 @@ +#!metalk8s_kubernetes + +apiVersion: v1 +kind: Service +metadata: + labels: + app: metalk8s-alert-logger + app.kubernetes.io/managed-by: salt + app.kubernetes.io/name: metalk8s-alert-logger + app.kubernetes.io/part-of: metalk8s + heritage: metalk8s + name: metalk8s-alert-logger + namespace: metalk8s-monitoring +spec: + ports: + - port: 19094 + protocol: TCP + targetPort: http + selector: + app: metalk8s-alert-logger + type: ClusterIP diff --git a/salt/metalk8s/deployed.sls b/salt/metalk8s/deployed.sls index 8310ea1418..ed61a5bbad 100644 --- a/salt/metalk8s/deployed.sls +++ b/salt/metalk8s/deployed.sls @@ -4,6 +4,7 @@ include: - metalk8s.kubernetes.coredns.deployed - metalk8s.repo.deployed - metalk8s.salt.master.deployed + - metalk8s.addons.alert-logger.deployed - metalk8s.addons.prometheus-operator.deployed - metalk8s.addons.nginx-ingress.deployed - metalk8s.addons.nginx-ingress-control-plane.deployed From 73f46cf1e28d0d1e31098a36b49e31acdfadc09b Mon Sep 17 00:00:00 2001 From: Alexandre Allard Date: Mon, 8 Mar 2021 15:56:16 +0100 Subject: [PATCH 08/17] salt: Improve fluent-bit parser Remove the date from the logs once parsed by fluent-bit as we do not need it anymore and it allows to improve the logs readability. Plus, it is also need by the alert history feature, this way we only end up with a JSON formatted alert in logs which makes it easier to parse/use by other components. Refs: #3180 --- .../logging/fluent-bit/deployed/configmap.sls | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls b/salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls index ab64bbe076..ec3ba907d7 100644 --- a/salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls +++ b/salt/metalk8s/addons/logging/fluent-bit/deployed/configmap.sls @@ -39,7 +39,7 @@ Create fluent-bit ConfigMap: Name tail Tag kube.* Path /var/log/containers/*.log - Parser docker + Parser container DB /run/fluent-bit/flb_kube.db Mem_Buf_Limit 5MB [INPUT] @@ -76,6 +76,10 @@ Create fluent-bit ConfigMap: Remove BOOT_ID Remove UID Remove GID + [FILTER] + Name modify + Match kube.* + Remove logtag {%- for index in range(loki.spec.deployment.replicas) %} [Output] Name loki @@ -109,7 +113,9 @@ Create fluent-bit ConfigMap: } parsers.conf: |- [PARSER] - Name docker - Format json + Name container + Format regex + Regex ^(?