Merge pull request #942 from joseph-v/ansible-installer

Add ansible delfin installer

installer/README.md

@@ -1,29 +1,71 @@
-# Introduction
-This is a standalone/non-containerized installer for SODA Infrastructure Manager (delfin) project.
-It contains a script and options to check the environment feasible for installing delfin. Installs required dependent software/binaries.
+# Delfin Installation Guide
+
+The SODA Delfin supports two types of installation
+* Installation using Ansible
+* Installation using Bash scripts
 
-# Supported OS
-Ubuntu 16.04, Ubuntu 18.04
+## Installation using Ansible
 
-# Prerequisite
+* Supported OS: **Ubuntu 20.04, Ubuntu 18.04**
+* Prerequisite: **Python 3.6 or above** should be installed
 
-- #### Ensure the logged-in user has root privileges.
+### Install steps
+
+```bash
+sudo apt-get update && sudo apt-get install -y git make curl wget libltdl7 libseccomp2 libffi-dev gawk
+git clone https://github.com/sodafoundation/delfin.git
+# git checkout <delfin-release-version>
+cd delfin/installer
+chmod +x install_dependencies.sh && ./install_dependencies.sh
+cd ansible
+export PATH=$PATH:/home/$USER/.local/bin
+sudo -E env "PATH=$PATH" ansible-playbook site.yml -i local.hosts -v
+```
+### Uninstall
+```bash
+sudo -E env "PATH=$PATH" ansible-playbook clean.yml -i local.hosts -v
+```
 
-- #### Setup Python3
-    Python3 and Pip3 should be installed on the system.
 
-    Note: If you don't have python3 in your system, you may follow below steps to setup python3 environment.
+### Logs
+Delfin processes execution logs can be found in /tmp/ folder
+* /tmp/api.log
+* /tmp/alert.log
+* /tmp/task.log
+* /tmp/exporter.log
+* /tmp/create_db.log
 
-    ```sh
-    apt-get install python3
-    apt-get install python3-pip
-    ```
+### How to use Delfin
+Delfin can be used either through dashboard or REST APIs.
+
+Please refer [user guides](https://docs.sodafoundation.io/guides/user-guides/delfin/dashboard/)
+
+
+
+## Installation using Bash Scripts
+This is a standalone/non-containerized installer for SODA Infrastructure Manager (delfin) project.
+It contains a script and options to check the environment feasible for installing delfin. Installs required dependent software/binaries.
 
-- #### Set PYTHONPATH to working directory
+* Supported OS: **Ubuntu 20.04, Ubuntu 18.04**
+* Prerequisite:
+  * **Python 3.6 or above** should be installed
+  * Ensure the logged-in user has **root privileges**.
+
+#### Installation steps
+```bash
+sudo -i
+apt-get install python3 python3-pip
+git clone https://github.com/sodafoundation/delfin.git && git checkout <delfin-release-version>
+cd delfin
+export PYTHONPATH=$(pwd)
+./installer/install
+```
+Refer below for installer options
 
-    ```sh
-    export PYTHONPATH=$(pwd)
-    ```
+#### Uninstall
+```bash
+./installer/uninstall
+```
 
 - #### [Optional] Setup Prometheus (for monitor performance metric through prometheus)
 
@@ -55,13 +97,13 @@ Ubuntu 16.04, Ubuntu 18.04
         ```sh
          root@root:/prometheus/prometheus-2.20.0.linux-amd64$ ./prometheus
         ```
-# Structure of the installer
+### Structure of the installer
 This installer comes with options of pre-check, install and uninstall
 pre-check: This script checks for the components required by delfin to function. If they are not present, precheck will install them.
 Install: Installs and starts the delfin process
 Uninstall: Uninstalls the delfin. Doesn't uninstall the required components. You may need to uninstall it explicitly using the native approach.
 
-# How to install
+### How to install
 To get help, execute 'install -h'. It will show help information
 
 Install script can be executed with three different switches to:
@@ -131,28 +173,12 @@ $ installer/install
 
 Note: Multiple instances of exporter and api is not allowed currently.
 
-#### Post install verification
-After delfin installation use the following command to verify all process 
-of delfin are running.
-```sh
-ps -aux | grep delfin
-```
-
-# Uninstall
-Running the uninstall script will stop all delfin processes and do cleanup
-```sh
-installer/uninstall
-
-# Example
-root@root1:~/delfin-demo/delfin$ installer/uninstall
-```
-
-# Logs
-All the installer logs are stored in the /var/log/sodafoundation directory.
+### Logs
+All the installer logs are stored in the /var/log/soda directory.
 The logs can be uniquely identified based upon the timestamp.
 
 
-# Test the running delfin setup/process
+## Test the running delfin setup/process
   1. Make sure all delfin process are up and running
      ```
      ps -ef|grep delfin
@@ -197,5 +223,5 @@ The logs can be uniquely identified based upon the timestamp.
 
      http://localhost:9090/graph
 
-# Limitation
+## Limitation
 Local installation, unlike Ansible installer, does not support SODA Dashboard integration.

installer/ansible/clean.yml

@@ -0,0 +1,27 @@
+# Copyright 2022 The SODA Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+# Defines some clean processes when banishing the nodes.
+- name: Cleanup delfin installation
+  hosts: 
+    - delfin-nodes
+  remote_user: root
+  vars_files:
+    - group_vars/delfin.yml
+  gather_facts: false
+  become: True
+  tasks:
+    - import_role:
+        name: cleaner

installer/ansible/group_vars/delfin.yml

@@ -0,0 +1,110 @@
+# Copyright 2022 The SODA Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+# Dummy variable to avoid error because ansible does not recognize the
+# file as a good configuration file when no variable in it.
+dummy:
+
+
+###########
+# GENERAL #
+###########
+
+# This field indicates local machine host ip
+host_ip: 127.0.0.1
+
+# delfin installation types are: 'repository', 'release' and 'container''
+delfin_installation_type: release
+
+# These fields below will specify the tag based on install_from type
+repo_branch: master
+release_version: v1.7.0
+
+# This field indicates which os family the system will be running, currently
+# support 'Debian' and 'RedHat'
+ansible_os_family: Debian
+
+# delete all source packages
+source_purge: true
+
+# delete database
+database_purge: true
+
+# Delfin  projects release versions
+delfin_release_version: v1.6.0
+
+# Depended  projects release versions
+dashboard_release_version: v1.7.0
+
+# URLs, Environment Variables, IP addresses and Ports list
+soda_delfin_url: "http://{{ host_ip }}:8190"
+
+# These fields are NOT suggested to be modified
+delfin_work_dir: /opt/delfin-linux-amd64
+delfin_config_dir: /etc/delfin
+venv: "{{ delfin_work_dir }}/venv"
+
+delfin_redis_ip: 127.0.0.1
+delfin_redis_port: 6379
+
+delfin_rabbitmq_user: delfinuser
+delfin_rabbitmq_pass: delfinpass
+
+# Configurable Perf collection interval in seconds
+performance_collection_interval: 900
+
+# Enable dynamic subprocess optimization for Perf collection
+enable_dynamic_subprocess: false
+
+# Exporter configurations for Kafka, Prometheus & Alert Manager
+# Uncomment exporters to enable
+performance_exporters: #PerformanceExporterPrometheus, PerformanceExporterKafka
+alert_exporters: #AlertExporterPrometheus
+
+# Exporter configurations for Kafka
+delfin_exporter_kafka_ip: 'localhost'
+delfin_exporter_kafka_port: 9092
+delfin_exporter_kafka_topic: 'delfin-kafka'
+
+# Exporter configurations for Prometheus
+delfin_exporter_prometheus_ip: 0.0.0.0
+delfin_exporter_prometheus_port: 8195
+delfin_exporter_prometheus_metrics_dir: '/var/lib/delfin/metrics'
+
+# Exporter configurations for Alert Manager
+delfin_exporter_alertmanager_host: 'localhost'
+delfin_exporter_alertmanager_port: 9093
+
+##############
+# REPOSITORY #
+##############
+
+# If user specifies intalling from repository, then he can choose the specific
+# repository branch
+delfin_repo_branch: "{{ repo_branch }}"
+
+# These fields are NOT suggested to be modified
+delfin_remote_url: https://github.com/sodafoundation/delfin.git
+
+###########
+# RELEASE #
+###########
+
+# If user specifies intalling from release,then he can choose the specific version
+delfin_release: "{{ delfin_release_version }}"
+
+# These fields are NOT suggested to be modified
+delfin_download_url: https://github.com/sodafoundation/delfin/archive/{{ delfin_release }}.tar.gz
+delfin_tarball_dir: /tmp/sodafoundation-delfin-{{ delfin_release }}-linux-amd64

installer/ansible/local.hosts

@@ -0,0 +1,16 @@
+# Copyright 2022 The SODA Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[delfin-nodes]
+localhost ansible_connection=local

installer/ansible/roles/cleaner/scenarios/delfin.yml

@@ -0,0 +1,68 @@
+# Copyright 2022 The SODA Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+- name: Stop delfin containers, if started
+  shell: "{{ item }}"
+  with_items:
+    - docker compose down
+  become: yes
+  ignore_errors: yes
+  args:
+    chdir: "{{ delfin_work_dir }}"
+
+- name: Get running delfin processes
+  shell: "ps -ef | grep -v grep | grep -i 'python3 /opt/delfin-linux-amd64/delfin/' | awk '{print $2}'"
+  register: running_processes
+
+- name: Kill running delfin processes
+  shell: "kill {{ item }}"
+  with_items: "{{ running_processes.stdout_lines }}"
+  ignore_errors: yes
+
+- wait_for:
+    path: "/proc/{{ item }}/status"
+    state: absent
+  with_items: "{{ running_processes.stdout_lines }}"
+  ignore_errors: yes
+  register: killed_processes
+
+- name: Force kill stuck processes
+  shell: "kill -9 {{ item }}"
+  with_items: "{{ killed_processes.results | select('failed') | map(attribute='item') | list }}"
+
+- name: Stop service of delfin, if started
+  service:
+    name: "{{ item }}"
+    state: stopped
+  with_items:
+    - rabbitmq-server
+    - redis-server
+  become: yes
+  ignore_errors: yes
+
+- name: clean up all delfin directories
+  file:
+    path: "{{ item }}"
+    state: absent
+    force: yes
+  with_items:
+    - "{{ delfin_work_dir }}"
+    - "{{ delfin_tarball_dir }}"
+  ignore_errors: yes
+  tags:
+    - delfin
+    - clean
+  when:
+    - source_purge is undefined or source_purge != false

installer/ansible/roles/cleaner/scenarios/release.yml

@@ -0,0 +1,24 @@
+# Copyright 2022 The SODA Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+- name: clean up all release files if installed from release
+  file:
+    path: "{{ item }}"
+    state: absent
+    force: yes
+  with_items:
+    - "{{ delfin_tarball_dir }}"
+  ignore_errors: yes
+  tags: clean