Skip to content

Installation

Jump to bottom
Chaganti, Sivakoteswararao edited this page Nov 20, 2019 · 2 revisions

Pipeline Engine Master

Master Image Build Instructions

Clone the POET Pipeline Library and follow the below instructions to build Pipeline Engine Master Docker Image.

cd POET-pipeline-library/master-image
docker-compose build --no-cache
docker images
docker login --username [USERNAME] --password [PASSWORD] [DOCKER_REGISTRY_SERVER]
docker push [DOCKER_IMAGE]

Please modify POET-pipeline-library/master-image/.env file with the Docker Tag to be generated.

DOCKER_ARTIFACTORY=<<DOCKER_REGISTRY>>
DOCKER_NAMESPACE=<<DOCKER_REPOSITORY>>
DOCKER_IMAGE=<<DOCKER_IMAGE_NAME>>
VERSION=<<DOCKER_TAG_VERSION>>

Agent Image Build Instructions

Clone the POET Pipeline Library and follow the below instructions to build Pipeline Engine Agent Docker Image.

cd POET-pipeline-library/agent-image
docker-compose build --no-cache
docker images
docker login --username [USERNAME] --password [PASSWORD] [DOCKER_REGISTRY_SERVER]
docker push [DOCKER_IMAGE]

Please modify POET-pipeline-library/agent-image/.env file with the Docker Tag to be generated.

DOCKER_ARTIFACTORY=<<DOCKER_REGISTRY>>
DOCKER_NAMESPACE=<<DOCKER_REPOSITORY>>
DOCKER_IMAGE=<<DOCKER_IMAGE_NAME>>
VERSION=<<DOCKER_TAG_VERSION>>

Mandatory Jenkins Plugins

The following Jenkins Plugins are bundled as part of Pipeline Engine Master Docker Image by default in the file POET-pipeline-library/master-image/lib/misc/plugins.txt. Modify the plugin version as needed.

  • cloudbees-folder:6.9

    This plugin allows users to create "folders" to organize jobs. Users can define custom taxonomies (e.g. by project type, organization type). Folders are nestable and you can define views within folders.

  • configuration-as-code:1.31

    The Configuration as Code plugin is used to configure Jenkins based on human-readable declarative configuration files. Writing such a file should be feasible without being a Jenkins expert, it is just translating into code a configuration process that is used for executing in the web UI.

  • credentials-binding:1.20

    This plugin allows credentials to be bound to environment variables for use from miscellaneous build steps.

  • git:3.12.1

    This plugin allows use of Git as a build SCM, including repository browsers for several providers.

  • job-dsl:1.76

    The job-dsl-plugin allows the programmatic creation of projects using a DSL. Pushing job creation into a script allows you to automate and standardize your Jenkins installation, unlike anything possible before.

  • kubernetes:1.19.3

    This plugin allows Jenkins agents to be dynamically provisioned on a Kubernetes cluster.

  • matrix-auth:2.4.2

    Offers matrix-based security authorization strategies (global and per-project). Matrix Authorization allows configuring the lowest level permissions, such as starting new builds, configuring items, or deleting them, individually.

  • workflow-aggregator:2.6

    A suite of plugins that lets you orchestrate automation, simple or complex.

Optional Jenkins Plugins

You can add additional plugins based on the organization requirement by modifying POET-pipeline-library/master-image/lib/misc/plugins.txt. Modify the plugin version as needed.

  • amazon-ecs:1.22
  • bitbucket-build-status-notifier:1.4.1
  • bitbucket-oauth:0.9
  • cloudbees-bitbucket-branch-source:2.4.6
  • docker-plugin:1.1.7
  • gitlab-plugin:1.5.12
  • ldap:1.20
  • splunk-devops:1.8.1
  • splunk-devops-extend:1.8.1
  • uno-choice:2.1

Jenkins Config as Code configuration files

Clone the POET Pipeline Library and update the jenkins plugins configurations that need to be changed.

The Files are located under the folder POET-pipeline-library/casc-jenkins-config-repo. Below are the default files required and you can add additional files based on other optional plugin that are required by the organization teams.

jcasc.yml
pipeline-CaC.yml
pipeline-JCasC.yml
pipeline-K8SCloudAgentsJCasC.yml

Once modified, these configurations need to be created as a separate repo to be used by Pipeline Engine Master Image at run time.

Deployment Instructions to Kubernetes Platform

You can run Pipeline Engine Master as either docker containers on standalone Virtual Machines or we can use Kubernetes as orchestration platform to spin up agents dynamically.

Clone the POET Pipeline Library and go to the folder POET-pipeline-library/kubernetes-deployment-configs. Below are the default files required and you can add additional files based on other requirements like Persistent Storage for Jenkins Home etc.

configmap.yml
deployment.yml
ingress.yml
service.yml
secrets.yml

Once modified, run the below kubectl commands to deploy the above configuration files to the cluster and namespace that user logged in.

kubectl apply -f configmap.yml
kubectl apply -f deployment.yml
kubectl apply -f ingress.yml
kubectl apply -f service.yml
kubectl apply -f secrets.yml
  • secrets.yml
    Set the below passwords based on the requirements. The passwords should be base64 encoded.
    CASC_JENKINS_CONFIG_REPO_PASSWORD - Required if the Jenkins Config as Code configuration files are pulled from remote repository at runtime while container is starting.
    PIPELINE_ENGINE_BACKUP_REPO_PASSWORD - Required if the Job configurations and next build numbers needs to be backed up to a remote repository.

Environment Variables

Set the below Environment variables in configmap.yml file.

Environment Variable Example Value Description
TZ America/Los_Angeles Define the Timezone used to run Pipeline Engine
PIPELINE_ENGINE_NAME {{PIPELINE_ENGINE_NAME}} Name of the Pipeline Engine to be provisioned.
KUBERNETES_CLUSTER {{KUBERNETES_CLUSTER}} Kubernetes Cluster where the Pipeline Engine is deployed.
KUBERNETES_NAMESPACE {{KUBERNETES_NAMESPACE}} Kubernetes Namespace where the Pipeline Engine is deployed.
KUBERNETES_APISERVER_URL {{KUBERNETES_APISERVER_URL}} Kubernetes API Server.
CASC_JENKINS_CONFIG /var/jenkins_home/casc_configs The Jenkins Configuration as Code plugin looks for the CASC_JENKINS_CONFIG environment variable to identify the Initial Jenkins Configuration to be loaded when starting Pipeline Engine
CASC_JENKINS_CONFIG_REPO github.com/{{CASC_JENKINS_CONFIG_REPO_USERNAME}}/casc-jenkins-config-repo.git The Github repository where Jenkins Config as Code Configuration files are located.
CASC_JENKINS_CONFIG_REPO_USERNAME {{CASC_JENKINS_CONFIG_REPO_USERNAME}} Username to download Jenkins Config as Code Configuration files.
CASC_JENKINS_CONFIG_PROFILE jcasc.yml Filename that contains the list of Jenkins Config as Code Configuration files to be downloaded.
MASTER_CPU {{MASTER_CPU}} CPU limit to be allocated to Master.
MASTER_MEMORY {{MASTER_MEMORY}} Memory limit to be allocated to Master.
AGENT_CPU {{AGENT_CPU}} CPU limit to be allocated to Agent.
AGENT_MEMORY {{AGENT_MEMORY}} Memory limit to be allocated to Agent.
AGENT_PORT {{AGENT_PORT}} The port that will be used to communicate between Master and Agents.
AGENT_EXECUTORS {{AGENT_EXECUTORS}} No.of Executors for agents.
AGENT_DOCKER_IMAGE {{AGENT_DOCKER_IMAGE}} Agent Docker Image.
HEALTH_URI /login HTTP URI that is used to perform the Healthcheck of the Pipeline Engine
HEALTH_STATUS_CODE 200 Response Code of Healthcheck to be validated
SERVER_PORT 8080 Server Port Pipeline Engine is listening inside container
JENKINS_ADMIN_ADDRESS {{JENKINS_ADMIN_ADDRESS}} set it as {{PIPELINE_ENGINE_NAME}}@{{JENKINS_URL}}
JENKINS_URL {{JENKINS_URL}} This Variable defines the Jenkins URL
PIPELINE_INTERNAL_TEMPLATES_REPO https://github.com/{{CASC_JENKINS_CONFIG_REPO_USERNAME}}/sample-os-templates.git The repository where shared Pipeline Engine Templates are defined to be used by all jobs
PIPELINE_INTERNAL_TEMPLATES_JENKINS_CREDENTIAL_ID {{PIPELINE_INTERNAL_TEMPLATES_JENKINS_CREDENTIAL_ID}} Jenkins Credential ID to clone Pipeline Templates at job run time
PIPELINE_GLOBAL_LIBRARY_URL https://github.com/tmobile/POET-pipeline-library.git POET Pipeline Library Repo
BACKUP_PIPELINE_ENGINE false BACKUP_PIPELINE_ENGINE Environment variable is identify whether the Job Configurations and Next Build Numbers need to be backup to a remote repository are not. Pipeline Engine Master image is already bundled with the scripts required to create a backup repo and upload job configurations to a Bitbucket Server.
RESTORE_PIPELINE_ENGINE false RESTORE_PIPELINE_ENGINE Environment variable is used to restore Jenkins Configuration in ${JENKINS_HOME} by pulling the remote changes from Pipeline Engine Backup Repo. The Conditional Restore is performed when RESTORE_PIPELINE_ENGINE Environment variable is defined and set to true every time Pipeline Engine is restarted.
PIPELINE_ENGINE_BACKUP_SERVER_URL {{PIPELINE_ENGINE_BACKUP_SERVER_URL}} Bitbucket Server where Pipeline Engine Backup repositories are created.
PIPELINE_ENGINE_BACKUP_PROJECT {{PIPELINE_ENGINE_BACKUP_PROJECT}} Bitbucket Project where Pipeline Engine Backup repositories are created
PIPELINE_ENGINE_BACKUP_REPO {{PIPELINE_ENGINE_BACKUP_REPO}} Pipeline Engine Backup Repo URL.
PIPELINE_ENGINE_BACKUP_REPO_USERNAME {{PIPELINE_ENGINE_BACKUP_REPO_USERNAME}} Username to create and upload job configs to Pipeline Engine Backup repositories
GIT_USER_NAME {{GIT_USER_NAME}} Git User Name
GIT_USER_EMAIL {{GIT_USER_EMAIL}} Git User Email

Table of contents

Clone this wiki locally