AdvantEDGE stopped working

[volm92]

Hi I'm Marco,
after using AdvantEDGE for a few days (from the graphical interface), it suddenly stopped working.
It no longer allowed me to deploy, and subsequently not even create a new sandbox.
I tried to delete all containers, pods and do the kubertenes installation procedure again and then dockerization and deployment but it doesn't work anymore.
How can I solve?
From which point can I start again to get it working again?
Thank you very much!

[volm92]

kubernetes*

[dilallkx]

Hi Marco @volm92,

Thank you for reaching out!
Could you please provide version information from your setup using the command meepctl version all?

Also, does your setup meet the minimum HW requirements documented here:
https://github.com/InterDigitalInc/AdvantEDGE/wiki/env-hw

If you observe an error during build or dockerization, please provide the meepctl verbose logs using meepctl -v <command>.

Thanks!

[dilallkx]

Ok, great!

One additional note... if you decide to reinstall docker on Ubuntu 20.04, you may need to specify slightly different package versions to install v19.03. Running apt-cache madison docker-ce should provide you with a list of available package versions for your distribution.

[volm92]

I am installing AEDGE on a new VM to keep it as a backup.
This time I use Ubuntu 18.04.
During the deploydep I got this error:

meepctl deploy dep
Using repo config file: /home/marco/AdvantEDGE/.meepctl-repocfg.yaml
Using meepctl config file: /home/marco/.meepctl.yaml
Using docker registry: meep-docker-registry:30001
Deployed meep-couchdb
Deployed meep-docker-registry
Deployed meep-grafana
Deployed meep-influxdb
Deployed meep-ingress
Deployed meep-kube-state-metrics
Deployed meep-open-map-tiles
Deployed meep-postgis
Error installing component [meep-prometheus]
Deployed meep-redis

The tool k9s is very useful! Thanks!

[dilallkx]

The Prometheus install is generally longer than the other dependencies and may take even more time on first install to pull the images. Could you please provide the deployment error logs using the command meepctl deploy dep -v?

[volm92]

meepctl deploy dep -v
Using repo config file: /home/marco/AdvantEDGE/.meepctl-repocfg.yaml
Using meepctl config file: /home/marco/.meepctl.yaml
Deploy called
[arg]  group: dep
[arg]  registry: 
[arg]  tag: latest
[flag] force: false
[flag] verbose: true
[flag] time: false
Using docker registry: meep-docker-registry:30001
Cmd: [mkdir -p /home/marco/.meep /home/marco/.meep/user /home/marco/.meep/user/values /home/marco/.meep/certs /home/marco/.meep/couchdb /home/marco/.meep/docker-registry /home/marco/.meep/grafana /home/marco/.meep/influxdb /home/marco/.meep/tmp /home/marco/.meep/virt-engine /home/marco/.meep/virt-engine/user-charts /home/marco/.meep/omt /home/marco/.meep/postgis /home/marco/.meep/prometheus /home/marco/.meep/prometheus/server /home/marco/.meep/prometheus/server/prometheus-db /home/marco/.meep/prometheus/alertmanager /home/marco/.meep/prometheus/alertmanager/alertmanager-db]
Result: OK
Cmd: [kubectl apply -f /home/marco/AdvantEDGE/charts/kube-prometheus-stack/crds/crd-servicemonitors.yaml]
Result: customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured

Cmd: [helm ls --filter meep-couchdb --short]
Result: 
Cmd: [helm install meep-couchdb --set fullnameOverride=meep-couchdb /home/marco/AdvantEDGE/charts/couchdb --replace --set persistentVolume.location=/home/marco/.meep/couchdb/]
Deployed meep-couchdb
Result: NAME: meep-couchdb
LAST DEPLOYED: Tue May 18 15:35:25 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Apache CouchDB is starting. Check the status of the Pods using:

  kubectl get pods --namespace default -l "app=couchdb,release=meep-couchdb"

Once all of the Pods are fully Ready, execute the following command to create
some required system databases:

  kubectl exec --namespace default meep-couchdb-couchdb-0 -c couchdb -- \
    curl -s \
    http://127.0.0.1:5984/_cluster_setup \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{"action": "finish_cluster"}'

Then it's time to relax.

Cmd: [sh -c /home/marco/AdvantEDGE/charts/docker-registry/create-k8s-ca-signed-cert.sh --certdir /home/marco/.meep/certs]
Result: creating certs in certdir /home/marco/.meep/certs 
Generating RSA private key, 2048 bit long modulus (2 primes)
.....+++++
...........................................+++++
e is 65537 (0x010001)
certificatesigningrequest.certificates.k8s.io "meep-docker-registry.default" deleted
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io/meep-docker-registry.default created
NAME                           AGE   SIGNERNAME                     REQUESTOR          CONDITION
meep-docker-registry.default   0s    kubernetes.io/legacy-unknown   kubernetes-admin   Pending
certificatesigningrequest.certificates.k8s.io/meep-docker-registry.default approved
W0518 15:35:26.137215    1094 helpers.go:553] --dry-run is deprecated and can be replaced with --dry-run=client.
secret/meep-docker-registry configured

Cmd: [helm ls --filter meep-docker-registry --short]
Result: 
Cmd: [helm install meep-docker-registry --set fullnameOverride=meep-docker-registry /home/marco/AdvantEDGE/charts/docker-registry --replace --set persistence.location=/home/marco/.meep/docker-registry/]
Deployed meep-docker-registry
Result: NAME: meep-docker-registry
LAST DEPLOYED: Tue May 18 15:35:26 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services meep-docker-registry)
  export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

Cmd: [helm ls --filter meep-grafana --short]
Result: 
Cmd: [helm install meep-grafana --set fullnameOverride=meep-grafana /home/marco/AdvantEDGE/charts/grafana --replace --set persistentVolume.location=/home/marco/.meep/grafana/ --set ingress.annotations.nginx\.ingress\.kubernetes\.io/auth-url=https://$http_host/auth/v1/authenticate?svc=grafana --set dashboards.default.runtime-environment-k8s-cluster.file=dashboards/runtime-environment-k8s-cluster.json --set dashboards.default.http-logs-aggregation.file=dashboards/http-loggers.json --set dashboards.default.http-single-log.file=dashboards/http-log-byId.json --set dashboards.default.network-metrics-aggregation.file=dashboards/network-metrics-aggregation.json --set dashboards.default.prometheus.file=dashboards/prometheus.json --set dashboards.default.runtime-environment-node.file=dashboards/runtime-environment-node.json --set dashboards.default.kubelet.file=dashboards/kubelet.json --set dashboards.default.network-metrics-point-to-point.file=dashboards/network-metrics-point-to-point.json --set dashboards.default.platform-advantedge.file=dashboards/platform-advantedge.json --set dashboards.default.platform-k8s.file=dashboards/platform-k8s.json]
Deployed meep-grafana
Result: coalesce.go:199: warning: destination for default is a table. Ignoring non-table value <nil>
NAME: meep-grafana
LAST DEPLOYED: Tue May 18 15:35:27 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace default meep-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   meep-grafana.default.svc.cluster.local

   If you bind grafana to 80, please update values in values.yaml and reinstall:
   
   securityContext:
     runAsUser: 0
     runAsGroup: 0
     fsGroup: 0

   command:
   - "setcap"
   - "'cap_net_bind_service=+ep'"
   - "/usr/sbin/grafana-server &&"
   - "sh"
   - "/run.sh"
   
   Details refer to https://grafana.com/docs/installation/configuration/#http-port.
   Or grafana would always crash.

   From outside the cluster, the server URL(s) are:
     http://


3. Login with the password from step 1 and the username: admin

Cmd: [helm ls --filter meep-influxdb --short]
Result: 
Cmd: [helm install meep-influxdb --set fullnameOverride=meep-influxdb /home/marco/AdvantEDGE/charts/influxdb --replace --set persistence.location=/home/marco/.meep/influxdb/]
Deployed meep-influxdb
Result: NAME: meep-influxdb
LAST DEPLOYED: Tue May 18 15:35:29 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
InfluxDB can be accessed via port 8086 on the following DNS name from within your cluster:

  http://meep-influxdb.default:8086

You can connect to the remote instance with the influx CLI. To forward the API port to localhost:8086, run the following:

  kubectl port-forward --namespace default $(kubectl get pods --namespace default -l app=meep-influxdb -o jsonpath='{ .items[0].metadata.name }') 8086:8086

You can also connect to the influx CLI from inside the container. To open a shell session in the InfluxDB pod, run the following:

  kubectl exec -i -t --namespace default $(kubectl get pods --namespace default -l app=meep-influxdb -o jsonpath='{.items[0].metadata.name}') /bin/sh

To view the logs for the InfluxDB pod, run the following:

  kubectl logs -f --namespace default $(kubectl get pods --namespace default -l app=meep-influxdb -o jsonpath='{ .items[0].metadata.name }')

Cmd: [helm ls --filter meep-ingress --short]
Result: 
Cmd: [helm install meep-ingress --set fullnameOverride=meep-ingress /home/marco/AdvantEDGE/charts/ingress-nginx --replace --set controller.service.ports.http=80 --set controller.hostPort.ports.http=80 --set controller.containerPort.http=80 --set controller.service.ports.https=443 --set controller.hostPort.ports.https=443 --set controller.containerPort.https=443]
Deployed meep-ingress
Result: NAME: meep-ingress
LAST DEPLOYED: Tue May 18 15:35:30 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
Get the application URL by running these commands:
  export POD_NAME=$(kubectl --namespace default get pods -o jsonpath="{.items[0].metadata.name}" -l "app=ingress-nginx,component=controller,release=meep-ingress")
  kubectl --namespace default port-forward $POD_NAME 8080:80
  echo "Visit http://127.0.0.1:8080 to access your application."

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

Cmd: [helm ls --filter meep-kube-state-metrics --short]
Result: 
Cmd: [helm install meep-kube-state-metrics --set fullnameOverride=meep-kube-state-metrics /home/marco/AdvantEDGE/charts/kube-state-metrics --replace]
Deployed meep-kube-state-metrics
Result: NAME: meep-kube-state-metrics
LAST DEPLOYED: Tue May 18 15:35:31 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
The exposed metrics can be found here:
https://github.com/kubernetes/kube-state-metrics/blob/master/docs/README.md#exposed-metrics

The metrics are exported on the HTTP endpoint /metrics on the listening port.
In your case, meep-kube-state-metrics.default.svc.cluster.local:8080/metrics

They are served either as plaintext or protobuf depending on the Accept header.
They are designed to be consumed either by Prometheus itself or by a scraper that is compatible with scraping a Prometheus client endpoint.

Cmd: [cp /home/marco/AdvantEDGE/charts/open-map-tiles/config.json /home/marco/.meep/omt/config.json]
Result: OK
Cmd: [helm ls --filter meep-open-map-tiles --short]
Result: 
Cmd: [helm install meep-open-map-tiles --set fullnameOverride=meep-open-map-tiles /home/marco/AdvantEDGE/charts/open-map-tiles --replace --set persistentVolume.location=/home/marco/.meep/omt/]
Deployed meep-open-map-tiles
Result: NAME: meep-open-map-tiles
LAST DEPLOYED: Tue May 18 15:35:32 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  http:///map
  http:///styles
  http:///images
  http:///fonts
  http:///data
  http:///leaflet-hash

Cmd: [helm ls --filter meep-postgis --short]
Result: 
Cmd: [helm install meep-postgis --set fullnameOverride=meep-postgis /home/marco/AdvantEDGE/charts/postgis --replace --set persistence.location=/home/marco/.meep/postgis/]
Deployed meep-postgis
Result: NAME: meep-postgis
LAST DEPLOYED: Tue May 18 15:35:33 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS name from within your cluster:

    meep-postgis.default.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace default meep-postgis -o jsonpath="{.data.postgresql-password}" | base64 --decode)

To connect to your database run the following command:

    kubectl run meep-postgis-client --rm --tty -i --restart='Never' --namespace default --image docker.io/postgis/postgis:12-3.0 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host meep-postgis -U postgres -d postgres -p 5432



To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/meep-postgis 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432

Cmd: [helm ls --filter meep-prometheus --short]
Result: 
Cmd: [helm install meep-prometheus --set fullnameOverride=meep-prometheus /home/marco/AdvantEDGE/charts/kube-prometheus-stack --replace --set alertmanager.alertmanagerSpec.securityContext.runAsUser=1000 --set prometheus.prometheusSpec.securityContext.runAsUser=1000 --set prometheus.prometheusSpec.persistentVolume.location=/home/marco/.meep/prometheus/server/ --set alertmanager.alertmanagerSpec.persistentVolume.location=/home/marco/.meep/prometheus/alertmanager/ --set nameOverride=prometheus]
Error installing component [meep-prometheus]
Result: Error: cannot re-use a name that is still in use

Cmd: [helm ls --filter meep-redis --short]
Result: 
Cmd: [helm install meep-redis --set fullnameOverride=meep-redis /home/marco/AdvantEDGE/charts/redis --replace]
Deployed meep-redis
Result: NAME: meep-redis
LAST DEPLOYED: Tue May 18 15:35:34 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
Redis can be accessed via port 6379 on the following DNS names from within your cluster:

meep-redis-master.default.svc.cluster.local for read/write operations
meep-redis-slave.default.svc.cluster.local for read-only operations



To connect to your Redis(TM) server:

1. Run a Redis(TM) pod that you can use as a client:
   kubectl run --namespace default meep-redis-client --rm --tty -i --restart='Never' \
   
   --image docker.io/bitnami/redis:6.0.9-debian-10-r66 -- bash

2. Connect using the Redis(TM) CLI:
   redis-cli -h meep-redis-master
   redis-cli -h meep-redis-slave

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/meep-redis-master 6379:6379 &
    redis-cli -h 127.0.0.1 -p 6379

[dilallkx]

You seem to have had some issues uninstalling Prometheus...

To show charts status: helm ls -a
To manually remove meep-prometheus: helm delete --no-hooks meep-prometheus

This should successfully remove meep-prometheus and allow you to run meepctl deploy dep afterwards.

[volm92]

Maybe I found the problem: resources get saturated because some pods are not closed when the simulation is closed and remain running using cpu and memory.
In fact I started AEDGE I created a SandBox called "aaa", then I typed meepctl delete core and meepctl delete dep and I closed.
Even when the pc is restarted, the pods referring to namespace "aaa" restart, thus consuming resources.
Is there a way to delete SandBox pods as well?

In the following image you can see k9s output and htop resources monitor.
Thanks

[volm92]

I reboot the VM, so something changed. The pods situation is this one as shown in the figure. In the .txt files the output of commands.

outputdescribe.txt
outputdf.txt
outputdpod.txt
outputdu.txt

[dilallkx]

Hi Marco,

The logs show that your node is experiencing Disk Pressure:

Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Fri, 21 May 2021 00:06:37 +0200   Fri, 21 May 2021 00:06:37 +0200   WeaveIsUp                    Weave pod has set this
  MemoryPressure       False   Fri, 21 May 2021 00:13:55 +0200   Thu, 29 Apr 2021 16:02:53 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         True    Fri, 21 May 2021 00:13:55 +0200   Fri, 21 May 2021 00:08:14 +0200   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure          False   Fri, 21 May 2021 00:13:55 +0200   Thu, 29 Apr 2021 16:02:53 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Fri, 21 May 2021 00:13:55 +0200   Wed, 19 May 2021 15:22:49 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled

This is not something we have observed much while using the platform, however more details can be found in the k8s documentation here:
https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/

Your ~/.meep/ folder usage seems fine, however your filesystem usage is quite high at 84%.

/dev/sda1       30830500 24549668   4691688  84% /

It seems like your VM disk size is set to ~32 Gi, which is low for running AdvantEDGE. We use persistent volumes that map to host paths. The persistent volume sizes are set as follows:

NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                             STORAGECLASS                      REASON   AGE
meep-couchdb-couchdb              10Gi       RWO            Retain           Bound    default/meep-couchdb-couchdb                      meep-couchdb-couchdb                       8d
meep-docker-registry              10Gi       RWO            Retain           Bound    default/meep-docker-registry                      meep-docker-registry                       8d
meep-grafana                      10Gi       RWO            Retain           Bound    default/meep-grafana                              meep-grafana                               8d
meep-influxdb                     8Gi        RWO            Retain           Bound    default/meep-influxdb                             meep-influxdb                              8d
meep-open-map-tiles               10Gi       RWO            Retain           Bound    default/meep-open-map-tiles                       meep-open-map-tiles                        8d
meep-postgis                      8Gi        RWO            Retain           Bound    default/meep-postgis                              meep-postgis                               8d
meep-prometheus-server            17Gi       RWO            Retain           Bound    default/pvc-prometheus-meep-prometheus-server-0   meep-prometheus-server                     8d
meep-virt-engine                  5Gi        RWO            Retain           Bound    default/meep-virt-engine                          meep-virt-engine                           8d
meep-virt-engine-user-values-pv   100Mi      RWO            Retain           Bound    default/meep-virt-engine-user-values-pvc          meep-virt-engine-user-values-sc            8d

This adds up to ~80Gi capacity, just for out platform volumes. The minimum capacity to support the platform should therefore be set to a value greater than this. Although it can/may work properly with less capacity, you may observe issues like the ones you are seeing for DiskPressure. Note that we recommend a 500GB storage in the hardware requirements section of the wiki:
https://github.com/InterDigitalInc/AdvantEDGE/wiki/env-hw

If you can't allocate this much for your VM I would recommend setting your disk size to at least 128Gi. Please try with a larger disk size and let us know if you continue to get evicted pods.

NOTE: The addition of new dependency pods in the more recent releases could explain why you did not observe these issues using a previous release.

Hope this fixes your issue!
Thanks!

[volm92]

Thank you very much, I'll proceed with the installation again in a VM with much more free disk space.
I will bother you again if I have any other problems!
Thank you very much, you have been very kind

[dilallkx]

Great! Please let us know if this resolves the issues you have observed.
Thanks!

[volm92]

Hello!
I installed a new VM with 150GB of free disk space. It seems to work! I used it a lot today and it still works ..
Thanks so much!

[volm92]

As shown in the image I have this problem. I don't remember if I solved it in the past and how.
However, looking for a solution, I found no solution. How can I solve? Thank you!

[roymx]

Hello @volm92 ... Since this discussion has already been solved, I opened a new discussion.
Here... #266