initial commit

.gitignore

@@ -0,0 +1 @@
+.vagrant/

README.md

@@ -0,0 +1,168 @@
+Jenkins Build Task Context Sharing
+==================================
+
+I was working with a team to make continuous integration happen, but not as an approved member of that team. Hence, I was reluctant to add technology, or make suggestions that would require debate.
+
+The project lead already had Jenkins set up, and a testing script which provisioned cloud resources and tested the web application. I was to create an AMI from the tested resource.
+
+I integrated this into the testing script, but it was deemed too complex. I would normally chain parameterized jobs together, but the plug-in was not installed. Also, it made sense to consider image-generation part of testing.
+
+So I devised a hack to share Jenkins context between shell tasks.
+
+The solution is to store important variables and their values in a text file in the build folder, and load these variables in another task environment where required. Example task scripts are stored in the default shared directory of the vagrant environment for simplicity.
+
+
+While I am sure this could be much improved by a bash scripting expert, or made "general", it suited the purpose.
+
+
+Testing with Vagrant
+--------------------
+
+First, launch and provision the jenkins environment:
+
+    $ cd jenkins_context/vagrant/jenkins
+    $ vagrant up
+
+You may then access Jenkins via http://192.168.111.99:8080/
+
+
+Job Testing
+-----------
+
+Explore the example_job and its configuration.
+
+Click `Build Now`, and then the link for #1 under Build History.
+
+Clicking Console Output, you should see something similar to this:
+
+```
+Started by user anonymous
+Building in workspace /var/lib/jenkins/workspace/example_job
+[example_job] $ /bin/sh -xe /tmp/hudson7227557787717119547.sh
++ /bin/bash -x /vagrant/test_01.sh
+++ dirname /vagrant/test_01.sh
++ . /vagrant/test_functions.sh
++ INSTANCE_ID=i-7c97aa82
++ KEY_PAIR_NAME=jenkins_testing
++ SECURITY_GROUP_ID=sg-7ba8b919
++ save_job_env i-7c97aa82 jenkins_testing sg-7ba8b919
++ INSTANCE_ID=i-7c97aa82
++ KEY_PAIR_NAME=jenkins_testing
++ SECURITY_GROUP_ID=sg-7ba8b919
+++ echo /var/lib/jenkins/jobs/example_job/builds/1/
++ my_directory=/var/lib/jenkins/jobs/example_job/builds/1/
+++ echo jenkins-example_job-1-env
++ env_file=jenkins-example_job-1-env
++ env_path=/var/lib/jenkins/jobs/example_job/builds/1/jenkins-example_job-1-env
++ echo INSTANCE_ID=i-7c97aa82
++ echo KEY_PAIR_NAME=jenkins_testing
++ echo SECURITY_GROUP_ID=sg-7ba8b919
+[example_job] $ /bin/sh -xe /tmp/hudson7426556713366106201.sh
++ /bin/bash -x /vagrant/test_02.sh
+++ dirname /vagrant/test_02.sh
++ . /vagrant/test_functions.sh
++ echo '========== before load ============'
+========== before load ============
++ env
+XDG_SESSION_ID=2
+HUDSON_SERVER_COOKIE=f8767e3d8fd2cc7b
+SHELL=/bin/bash
+TERM=unknown
+BUILD_TAG=jenkins-example_job-1
+WORKSPACE=/var/lib/jenkins/workspace/example_job
+USER=jenkins
+JENKINS_HOME=/var/lib/jenkins
+PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+MAIL=/var/mail/jenkins
+PWD=/var/lib/jenkins/workspace/example_job
+LANG=en_US.UTF-8
+JOB_NAME=example_job
+BUILD_DISPLAY_NAME=#1
+BUILD_ID=2014-08-25_23-31-24
+HOME=/var/lib/jenkins
+SHLVL=2
+EXECUTOR_NUMBER=0
+JENKINS_SERVER_COOKIE=f8767e3d8fd2cc7b
+NODE_LABELS=master
+LOGNAME=jenkins
+HUDSON_HOME=/var/lib/jenkins
+NODE_NAME=master
+BUILD_NUMBER=1
+XDG_RUNTIME_DIR=/run/user/1000
+HUDSON_COOKIE=d9c20b74-8945-4df7-a333-65b8a73535c5
+_=/usr/bin/env
++ echo ==================================
+==================================
++ load_job_env
+++ echo /var/lib/jenkins/jobs/example_job/builds/1/
++ my_directory=/var/lib/jenkins/jobs/example_job/builds/1/
+++ echo jenkins-example_job-1-env
++ env_file=jenkins-example_job-1-env
++ env_path=/var/lib/jenkins/jobs/example_job/builds/1/jenkins-example_job-1-env
++ read line
++ export INSTANCE_ID=i-7c97aa82
++ INSTANCE_ID=i-7c97aa82
++ read line
++ export KEY_PAIR_NAME=jenkins_testing
++ KEY_PAIR_NAME=jenkins_testing
++ read line
++ export SECURITY_GROUP_ID=sg-7ba8b919
++ SECURITY_GROUP_ID=sg-7ba8b919
++ read line
++ echo '========== after load ============'
+========== after load ============
++ env
+XDG_SESSION_ID=2
+HUDSON_SERVER_COOKIE=f8767e3d8fd2cc7b
+SHELL=/bin/bash
+TERM=unknown
+BUILD_TAG=jenkins-example_job-1
+SECURITY_GROUP_ID=sg-7ba8b919
+WORKSPACE=/var/lib/jenkins/workspace/example_job
+USER=jenkins
+JENKINS_HOME=/var/lib/jenkins
+PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+MAIL=/var/mail/jenkins
+KEY_PAIR_NAME=jenkins_testing
+PWD=/var/lib/jenkins/workspace/example_job
+LANG=en_US.UTF-8
+JOB_NAME=example_job
+BUILD_DISPLAY_NAME=#1
+BUILD_ID=2014-08-25_23-31-24
+HOME=/var/lib/jenkins
+SHLVL=2
+EXECUTOR_NUMBER=0
+JENKINS_SERVER_COOKIE=f8767e3d8fd2cc7b
+NODE_LABELS=master
+LOGNAME=jenkins
+HUDSON_HOME=/var/lib/jenkins
+NODE_NAME=master
+BUILD_NUMBER=1
+XDG_RUNTIME_DIR=/run/user/1000
+HUDSON_COOKIE=d9c20b74-8945-4df7-a333-65b8a73535c5
+INSTANCE_ID=i-7c97aa82
+_=/usr/bin/env
++ echo ==================================
+==================================
+Finished: SUCCESS
+```
+
+While it may not seem miraculous, we have made INSTANCE_ID available to an independent task shell, which is normally impossible.
+
+If I am misguided and there is a standard way to do this in Jenkins, please send me an email. I will update this document with your corrections.
+
+
+References
+----------
+
+I have included references for Jenkins pipelining and launching standard parameterized builds via web POST. We used web calls to launch a job to purge cloud resources -- but the technique is not a replacement for build pipelines (which clearly track status for every related task).
+
+* [How to use Jenkins for Job Chaining and Visualizations](http://zeroturnaround.com/rebellabs/how-to-use-jenkins-for-job-chaining-and-visualizations/)
+* [Parameterized Build (Jenkins Wiki)](https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build)
+* [Jenkins: The Definitive Guide](http://shop.oreilly.com/product/0636920010326.do)
+
+
+Acknowledgments
+---------------
+
+The basic project structure follows that devised by my teammate @elasticdog (author of transcrypt). I choose to keep the structure, as it emphasizes the technologies involved.

ansible/playbooks/dev_jenkins.yml

@@ -0,0 +1,11 @@
+---
+# Deploy a dev environment for working on Jenkins CI.
+
+- hosts:
+  - development
+  roles:
+    - role: common
+      tags: common
+
+    - role: jenkins
+      tags: jenkins

ansible/roles/common/tasks/main.yml

@@ -0,0 +1,11 @@
+---
+
+- name: ensure the apt package cache is up-to-date
+  sudo: true
+  apt: update_cache=yes cache_valid_time=3600
+  register: apt_cache_update
+
+- name: remove orphaned dependency packages
+  sudo: true
+  command: /usr/bin/apt-get --assume-yes autoremove
+  changed_when: false

ansible/roles/java/tasks/main.yml

@@ -0,0 +1,5 @@
+---
+
+- name: install the openjdk (java development kit) package
+  sudo: true
+  apt: name=openjdk-7-jdk state=present

ansible/roles/jenkins/README.md

@@ -0,0 +1,33 @@
+Jenkins CI
+==========
+
+An extensible open source continuous integration server.
+
+Testing with Vagrant
+--------------------
+
+Jenkins supports an architecture consisting of a master server with additional build slaves. This Jenkins playbook only configures the master server with default settings.
+
+Adding New Jobs
+---------------
+
+Jenkins uses XML files to define jobs. It's easiest to use the web interface to configure your new jobs, and then copy their configuration files from your VM to the local host before checking them into Git.
+
+1. Copy the *config.xml* file for your job to the local host
+2. Rename it to have the title of your job and give it the extension *.xml.j2*
+3. Put the file under the *roles/jenkins/templates/jobs/* directory
+4. Ansible will handle the rest during provisioning
+
+For example, if you've launched the *jenkins* VM and created a job named "new-job", you could copy it off of the VM and put it into the proper place with:
+
+    $ cd infrastructure/vagrant/jenkins/
+    $ vagrant ssh
+    $ cd /var/lib/jenkins/jobs/new-job
+    $ more config.xml
+
+Then copy the contents of config.xml to your job template, inserting variables and includes where appropriate.
+
+References
+----------
+
+* [Upstream Documentation](http://jenkins-ci.org/)

ansible/roles/jenkins/defaults/main.yml

@@ -0,0 +1,45 @@
+---
+
+# jenkins home directory
+jenkins_home: /var/lib/jenkins
+
+# local path to the updates center metadata file
+jenkins_updates_file: "{{ jenkins_home }}/updates/default.json"
+
+# remote url of the updates center metadata file
+jenkins_updates_url: "http://updates.jenkins-ci.org/update-center.json"
+
+# path to the jenkins cli jar file
+jenkins_cli_jar: /usr/share/jenkins/jenkins-cli.jar
+
+# number of seconds to wait for jenkins to come up
+jenkins_delay_seconds: 120
+
+# jenkins http port
+# keep standard port, but when using web server, it helps to use
+# variables for port and listening-ip
+jenkins_http_port: 8080
+
+# command for running jenkins cli tasks
+jenkins_cli_cmd: "/usr/bin/java -jar {{ jenkins_cli_jar }} -s http://localhost:{{ jenkins_http_port }}/"
+
+# plugins are not required by the demo code, but here are some
+# typical removals and additions, alter to taste
+
+# jenkins plugins to disable
+jenkins_disabled_plugins:
+  - ant
+  - cvs
+  - ldap
+  - maven-plugin
+  - subversion
+  - translation
+  - windows-slaves
+
+# jenkins plugins to install
+jenkins_required_plugins:
+  - build-pipeline-plugin
+  - copyartifact
+  - ghprb
+  - git
+  - github

ansible/roles/jenkins/handlers/main.yml

@@ -0,0 +1,7 @@
+---
+
+- name: reload jenkins configuration
+  command: "{{ jenkins_cli_cmd }} reload-configuration"
+
+- name: safely restart the jenkins daemon
+  command: "{{ jenkins_cli_cmd }} safe-restart"

ansible/roles/jenkins/meta/main.yml

@@ -0,0 +1,4 @@
+---
+
+dependencies:
+  - { role: java, tags: java }

ansible/roles/jenkins/tasks/jobs.yml

@@ -0,0 +1,18 @@
+---
+# Configure the Jenkins jobs and dependencies.
+
+- name: add job directories
+  sudo: true
+  file: path={{ jenkins_home }}/jobs/{{ item | basename | replace(".xml.j2", "") }} state=directory
+    owner=jenkins group=nogroup mode=0755
+  with_fileglob:
+    - ../templates/jobs/*
+
+- name: add job configs
+  sudo: true
+  template: src={{ item }}
+    dest={{ jenkins_home }}/jobs/{{ item | basename | replace(".xml.j2", "") }}/config.xml
+    owner=jenkins group=nogroup mode=0644
+  with_fileglob:
+    - ../templates/jobs/*
+  notify: reload the jenkins configuration

ansible/roles/jenkins/tasks/main.yml

@@ -0,0 +1,30 @@
+---
+
+- name: add jenkins deb repository key to apt
+  sudo: true
+  apt_key: url=http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key state=present
+
+- name: add the jenkins deb repository as an apt source
+  sudo: true
+  apt_repository: repo='deb http://pkg.jenkins-ci.org/debian binary/'
+    state=present update_cache=yes
+
+- name: install the jenkins package
+  sudo: true
+  apt: name=jenkins state=present
+  register: jenkins_install
+
+- name: wait for jenkins daemon to accept connections
+  wait_for: port={{ jenkins_http_port }} delay={{ jenkins_delay_seconds }} state=started
+  when: jenkins_install.changed
+
+- name: download jenkins cli jar
+  sudo: true
+  get_url: url=http://{{ ansible_ssh_host }}:{{ jenkins_http_port }}/jnlpJars/jenkins-cli.jar dest={{ jenkins_cli_jar }}
+    owner=root group=root mode=0644
+
+# plugins are not relevant to demo code, but appear in the references
+# about creating job pipelines, so ¯\_(ツ)_/¯
+
+# - include: plugins.yml
+- include: jobs.yml

ansible/roles/jenkins/tasks/plugins.yml

@@ -0,0 +1,52 @@
+---
+# Plugin Installation
+
+# ---
+# work around update center metadata bug
+# https://issues.jenkins-ci.org/browse/JENKINS-10061
+# ---
+- name: create jenkins updates directory
+  sudo: true
+  file: path={{ jenkins_home }}/updates state=directory
+    owner=jenkins group=nogroup mode=0755
+
+# must remove javascript wrapper...
+- name: download updates center metadata file
+  sudo: true
+  shell: /usr/bin/curl --location {{ jenkins_updates_url }} | /bin/sed '1d;$d' > {{ jenkins_updates_file }}
+    creates={{ jenkins_updates_file }}
+
+- name: fix updates center metadata permissions
+  sudo: true
+  file: path={{ jenkins_updates_file }} state=file
+    owner=jenkins group=nogroup mode=0644
+# ---
+# ---
+
+- name: check for existing plugins
+  command: "{{ jenkins_cli_cmd }} list-plugins"
+  register: jenkins_existing_plugins
+  changed_when: false
+
+- name: install required jenkins plugins
+  command: "{{ jenkins_cli_cmd }} install-plugin {{ item }} -deploy"
+  with_items: jenkins_required_plugins
+  when: jenkins_existing_plugins.stdout.find("{{ item }}") == -1
+
+# obsolete plugins are listed with the new version in parentheses
+- name: check for outdated plugins
+  shell: /usr/bin/java -jar {{ jenkins_cli_jar }} -s http://localhost:8080/ list-plugins | awk '/\)$/{ print $1 }'
+  register: jenkins_outdated_plugins
+
+# updates to plugins won't actually be applied until a restart
+- name: update outdated plugins
+  command: "{{ jenkins_cli_cmd }} install-plugin {{ item }}"
+  with_items: jenkins_outdated_plugins.stdout_lines
+  notify: safely restart the jenkins daemon
+
+- name: disable unneccessary plugins
+  sudo: true
+  copy: dest={{ jenkins_home }}/plugins/{{ item }}.jpi.disabled content=""
+    owner=jenkins group=nogroup mode=0644
+  with_items: jenkins_disabled_plugins
+  notify: safely restart the jenkins daemon