docker-compose development environment : Jenkins custom Docker image + Atlassian Bitbucket + Sonarqube + etc.
This repo contains a docker/docker-compose configuration for Jenkins custom Docker image + Sonarqube && Atlassian Bitbucket for development environment setup. Note: Sonarqube & Bitbucket applications comes with it's own PostgreSQL instances (docker containers: postgres for sonarqube && atl_database for BB database), user, database and docker volume.
git, docker and docker-compose to be installed.
$ sh ./utils/docker-setup.sh
Custom Jenkins image is built with the following features:
- Preinstalled plugins
- Default administrator
- No setup wizard
- Default location configuration
- Default Maven installation
- Capability of running Ansible playbooks
- Docker in Docker
- Adding a global credentials
- Adding an AWS credentials
- In-process Script Approval
- Integration with Bitbucket
- Integration with SonarQube
- Integration with Slack
- JVM Metrics
- UI tests capability
- Jenkins hardening
Note: Set some secure passwords in .env and pg-init-scripts/init.sql if you needed.
During configuration of the applications via the setup procedure, use the following database settings:
host: atl_database
port: 5432
database: <app>db
database user: <app>user
database user password: <password specified in init.sql>
To fully clean reset an application, you have to delete the associated docker volume and clean the associated database.
$ sudo sysctl -w vm.max_map_count=262144 // Note: For sonarqube to work properly
$ docker-compose up -d
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "jenkins-dev-environment_default" with the default driver
Creating postgres ... done
Creating atl_database ... done
Creating bitbucket ... done
Creating sonarqube ... done
Creating jenkins-dev ... done
$ docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------------------
atl_database docker-entrypoint.sh postgres Up 0.0.0.0:5431->5432/tcp
bitbucket /usr/bin/tini -- /entrypoi ... Up 0.0.0.0:7990->7990/tcp, 0.0.0.0:7999->7999/tcp
jenkins-dev /entrypoint.sh Up 50000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8081->8081/tcp
postgres docker-entrypoint.sh postgres Up (healthy) 5432/tcp
sonarqube ./bin/run.sh Up 0.0.0.0:9000->9000/tcp
$ docker-compose logs sonarqube|tail
sonarqube | 2021.07.21 15:37:49 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
sonarqube | 2021.07.21 15:37:49 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
sonarqube | 2021.07.21 15:37:54 INFO ce[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
sonarqube | 2021.07.21 15:37:54 INFO ce[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://postgres/sonar
sonarqube | 2021.07.21 15:38:07 INFO ce[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
sonarqube | 2021.07.21 15:38:08 INFO ce[][o.s.c.c.CePluginRepository] Load plugins
sonarqube | 2021.07.21 15:38:21 INFO ce[][o.s.c.c.ComputeEngineContainerImpl] Running Community edition
sonarqube | 2021.07.21 15:38:22 INFO ce[][o.s.ce.app.CeServer] Compute Engine is operational
sonarqube | 2021.07.21 15:38:22 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
sonarqube | 2021.07.21 15:38:22 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
| Application | URL |
|---|---|
| Bitbucket | http://localhost:7990/ |
| Jenkins | http://localhost:8080/ |
The PostgreSQL database is only accessible from inside the cluster on port 5432. |
docker-compose down
docker volume ls|grep jenkins-dev-environment|awk '{print $2}'|xargs docker volume rm -
A bunch of plugins are installed during the image build. Once they increase the image size considerably, the Alpine version of the Jenkins Docker image is used as the base image (jenkins/jenkins:lts-alpine), in order to keep the built image as small as possible.
Most of the plugins enable other features, as described below.
The default Jenkins administrator account is created during the execution of default-user.groovy. Its credentials are obtained from the environment variables:
- JENKINS_USER (default: admin)
- JENKINS_PASS (default: jenkins)
With preinstalled plugins and a default administrator, there is no need of following the Jenkins wizard during its setup. For this reason, the wizard is disabled: -Djenkins.install.runSetupWizard=false.
The default Jenkins location is configured during the execution of config-location.groovy. The Jenkins URL and the e-mail address Jenkins use as the sender for e-mail notification are obtained from the environment variables:
- JENKINS_URL (default: http://localhost:8080)
- JENKINS_EMAIL (default: [email protected])
The default Apache Maven installation is configured during the execution of config-maven.groovy. The Maven version is obtained from the environment variable:
- MAVEN_VERSION (default: 3.6.3)
Maven can then be referenced by M3 in the Jenkinsfile, like in the example below:
node {
def mvnHome = tool 'M3'
...
stage('Build and Unit Tests') {
sh "'${mvnHome}/bin/mvn' clean install"
}
...
}The default Ansible installation is configured during the execution of config-ansible.groovy. Ansible can then be referenced by Ansible in the Jenkinsfile, like in the example below:
node {
...
stage('Deploy to AWS') {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws', accessKeyVariable: 'AWS_ACCESS_KEY_ID', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
ansiblePlaybook(installation: 'Ansible', playbook: 'deploy-to-aws.yml')
}
}
...
}This feature is enabled by the Ansible plugin.
The jenkins Docker image is prepared itself to enable the execution of Docker commands. So, you are able to run pipelines in the jenkins Docker container that build Docker images, push Docker images to a Docker Registry or execute any other Docker command (example below). The only requirement is to bind mount your host Docker daemon Unix socket to the container Docker daemon Unix socket: -v /var/run/docker.sock:/var/run/docker.sock.
node {
def appDockerImage
...
stage('Build Docker Image') {
appDockerImage = docker.build("davarski/webapp-docker")
}
stage('Deploy Docker Image') {
docker.withRegistry("", "dockerhub") {
appDockerImage.push()
}
}
...
}Note: See EXAMPLE
A new global credentials can be created in Jenkins with the execution of add-credentials.groovy. It only happens if the following environment variables are defined:
- CREDENTIALS_ID
- CREDENTIALS_USER
- CREDENTIALS_PASS
The global credentials can then be referenced by its credentialsId in the Jenkinsfile, like in the example below:
node {
...
stage('Deploy Java Artifacts') {
withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: 'ossrh', usernameVariable: 'OSSRH_USER', passwordVariable: 'OSSRH_PASSWORD']]) {
sh "'${mvnHome}/bin/mvn' -s .travis.settings.xml source:jar deploy -DskipTests=true"
}
}
...
}A new AWS credentials can be created in Jenkins with the execution of add-aws-credentials.groovy. It only happens if the following environment variables are defined:
- AWS_CREDENTIALS_ID
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
The AWS credentials can then be referenced by its credentialsId in the Jenkinsfile, like in the example below:
node {
...
stage('Deploy to AWS') {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws', accessKeyVariable: 'AWS_ACCESS_KEY_ID', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
ansiblePlaybook(installation: 'Ansible', playbook: 'deploy-to-aws.yml')
}
}
...
}This feature is enabled by the CloudBees AWS Credentials plugin.
Ref: AWS example -> https://github.com/adavarski/DevOps-AWS-demo
Note: Use Dockerfile-TF for jenkins custom image
...
String awsCredentialsId = 'AWS-demo'
...
stage('TF init') {
steps {
dir('aws-tf/Jenkins-EC2') {
script {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: awsCredentialsId,
accessKeyVariable: 'AWS_ACCESS_KEY_ID',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']])
{
sh "terraform init"
}
}
}
}
}
stage('TF apply env') {
when{ equals expected: "CREATE", actual: "${params.Action}"}
steps {
dir('aws-tf/Jenkins-EC2') {
script {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: awsCredentialsId,
accessKeyVariable: 'AWS_ACCESS_KEY_ID',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']])
{
sh "terraform apply -auto-approve"
}
}
}
}
}
...
This feature is enabled by the CloudBees AWS Credentials plugin.
An in-process script or a method signature can be approved in Jenkins with the execution of approve-signature.groovy. It only happens if the following environment variable is defined:
- SIGNATURE
Jenkins can be integrated to Bitbucket with the execution of config-bitbucket.groovy. It only happens if the following environment variables are defined:
- BITBUCKET_URL - the endpoint where your Bitbucket instance is available
- BITBUCKET_CREDENTIALS_ID - the global credentials id previously added to Jenkins with the Bitbucket username and password
This feature is enabled by the Bitbucket plugin.
For better integration you can install Bitbucket Server integration plugin: See EXAMPLE using Bitbucket Server integration plugin.
Jenkins can be integrated to SonarQube with the execution of config-sonarqube.groovy. It only happens if the following environment variable is defined:
- SONARQUBE_URL - the endpoint where your SonarQube instance is available
SonarQube can then be referenced by SonarQube in the Jenkinsfile, like in the example below:
node {
...
stage('Static Code Analysis') {
withSonarQubeEnv('SonarQube') {
sh "'${mvnHome}/bin/mvn' sonar:sonar"
}
}
...
}ℹ️ You don't need to pass the installationName parameter to withSonarQubeEnv if just one SonarQube server was configured in Jenkins. More details in Using a Jenkins pipeline.
This feature is enabled by the SonarQube plugin.
Jenkins can be integrated to Slack with the execution of config-slack.groovy. It only happens if the following environment variables are defined:
- SLACK_WORKSPACE
- SLACK_TOKEN
Slack can then be used in the Jenkinsfile, like in the example below:
node {
...
stage('Results') {
...
slackSend(channel: 'builds', message: 'Pipeline succeed!', color: 'good')
}
}This feature is enabled by the Slack Notification plugin.
The Jenkins JVM metrics are exposed by jmx_exporter, a process for exposing JMX Beans via HTTP for Prometheus consumption. The JVM metrics are exposed through port 8081, as passed to the Java Agent:
-javaagent:/usr/bin/jmx_exporter/jmx_prometheus_javaagent.jar=8081:/usr/bin/jmx_exporter/config.yaml
The Jenkins image has installed Firefox ESR and geckodriver (available in /usr/local/bin/geckodriver), enabling that way UI tests with Selenium.
The example below shows how the UI test can be performed during the execution of your Jenkinsfile:
node {
...
stage('UI Tests') {
sh "'${mvnHome}/bin/mvn' -f test-selenium test -Dwebdriver.gecko.driver=/usr/local/bin/geckodriver -Dheadless=true"
}
...
}The Jenkins security is improved during the execution of harden-jenkins.groovy and default-project-authorization.groovy, when the following actions are taken:
- Enabling CSRF protection;
- Enabling Agent -> Master access control;
- Disabling the deprecated JNLP;
- Preventing builds run as SYSTEM by configuring access control for builds.