Thali Continuous Integration with Jenkins

One of my first projects with my new team at Microsoft was to setup a continous build/integration service for Thali. The build system would have to:

  1. Test pull requests (PR). Whenever a PR is opened against any of the Thali projects, we want to build it, test it, and update the status of the PR.
  2. Publish blessed binaries to repository manager. Commits to the repo are built, and if the tests are successful, publish the artifacts to a public repository.

Which translated to the following technical requirements:

  • Ability to build Thali on multiple platforms (Mac, Linux, Windows, Android).
  • Support retrieval of source code via Git, building via Gradle, and publishing of binaries to Artifactory via Maven.
  • Most importantly, free.

The Wikipedia page has a great table comparing all the CI options. Only one of them fit the bill for Thali: Jenkins. Jenkins is built and since it's split with Hudson, has garnered a large user community. Due to its plugin architecture, Jenkins supports a wide variety of version control and build systems.

This blog post will provide a walkthrough of:

  1. Installing and maintaining Jenkins on Azure
  2. Setting up a multiplatform (distributed) build with Azure slaves

Jenkins on Azure

Installation

A pre-built Jenkins image is available on VM Depot that we can use to deploy to our Azure VM. The instructions for doing so can be found here. Once installed, the Jenkins portal will be available at:

www.[your-url-here].cloudapp.net/jenkins  

With the default login credentials being:

  • username: user
  • password: bitnami

Managing Jenkins

With the instance being an Azure virtual machine, we can login to the machine via SSH. In order to do so, we need to enable port 22 on the Azure Management Portal.

Once enabled, we can SSH to the machine: ssh [user]@[your-url-here].cloudapp.net and manage the Jenkins/Bitnami service directly via the service command: sudo service bitnami [start/stop/restart].

Updates to Jenkins are pushed out quite frequently; they have two release tracks:

  1. bleeding edge - bug fixes are pushed out weekly.
  2. stability - for those that don't want to be working on the latest and greatest, a more stable build is pushed out every 12 weeks

To update the Jenkins package:

# download the latest WAR and replace it with existing
cd /home/user/stack/apache-tomcat/webapps  
sudo wget http://updates-jenkins-ci.org/download/war/[version]/jenkins.war  

Multi-Platform Builds

The overall build workflow will be as follows:

  1. GitHub webhooks will notify the Jenkins master node of a change (e.g. commit, PR) to the repository
  2. The Azure Slave plugin running on the Jenkins master node will provision a Windows and Linux VM (if necessary)
  3. The Jenkins master node will coordinate and communicate to the slaves via JNLP or SSH. Each slave will be instructed to:
    • fetch source using Git
    • build using Gradle
    • publish artifacts to the Artifactory.

Plugins

We are going to leverage the following plugins:

Installation

Install plugins from the plugin manager: [your-url-here].cloudapp.net/jenkins/pluginManager/available.

If the list is empty, you can force Jenkins to check for updates under the "Advanced" tab.

Configuration

Configurations for all plugins are found under the main Jenkins configuration page: Manage Jenkins -> Configure or [your-url-here].cloudapp.net/jenkins/configure.

Azure Slave Plugin

This blog post will help walkthrough configuring the plugin.

  • Image Family or ID: specifies the image to be used when provisioning a new VM. In the case of Thali, we created custom Ubuntu and Window images with the build pre-requisites (Java, Android SDK, git, etc) already installed.
  • Launch Method: Generally, use SSH for a Linux image. JNLP for a Windows image. See section below for more specifics.
  • Init Script: specifics the script that will be launched once the machine starts up. The plugin makes use of the Custom Script Execution for Windows and Cloud-Init for Linux to execute the scripts. Here are the scripts that I used for Linux and Windows.

Configure JNLP (if using Windows Slaves)

By default, Jenkins uses SSH to communicate with the slave machines, as there is no SSH server pre-installed on Windows, we'll use JNLP instead.

Configure the Jenkins master to use a fixed port for JNLP. Manage Jenkins -> Configure Global Security or [your-url-here].cloudapp.net/jenkins/configureSecurity.

Ensure the same port number is enabled on the Azure Management Portal for the Jenkins master VM by creating an endpoint.

Windows Azure slaves will need to:

Here's an init script that will achieve just that:

# https://gist.github.com/jpoon/39ab4f3692147758d629
#
# Windows Image already has preinstalled:
#  1) Java JDK 8u5
#     $env:JAVA_HOME = C:\Program Files (x86)\Java\jdk1.8.0_05
#     $env:PATH = $env:PATH;$env:JAVA_HOME
#  2) Visual Studio 2013 Express for Desktop
#  3) Android SDK 19. Build Tools 19.0.1, 19.0.3

Write-Host 'Begin Init Script';

Write-Host "Arguments: $args";  
$jenkinsServerUrl = $args[0];
$slaveVmName = $args[1];

Write-Host 'Downloading Jenkins slave.jar...';  
md -Path 'c:\jenkins' -Force  
$slaveSource =  $jenkinsServerUrl + 'jnlpJars/slave.jar';
$slaveJarFilePath = 'c:\jenkins\slave.jar';
$wc = New-Object System.Net.WebClient;
$wc.DownloadFile($slaveSource, $slaveJarFilePath);

Write-Host 'Executing Jenkins slave.jar...';  
$javaExe = $env:JAVA_HOME + '\bin\java.exe';
$jnlpUrl = $jenkinsServerUrl + 'computer/' + $slaveVmName + '/slave-agent.jnlp';
$credentials ='<INSERT YOUR OWN CREDENTIALS HERE>';

$process = New-Object System.Diagnostics.Process;
$process.StartInfo.FileName = $javaExe;
$process.StartInfo.Arguments = "-jar $slaveJarFilePath -jnlpCredentials $credentials -jnlpUrl $jnlpUrl"
$process.StartInfo.RedirectStandardError = $true;
$process.StartInfo.RedirectStandardOutput = $true;
$process.StartInfo.UseShellExecute = $false;
$process.StartInfo.CreateNoWindow = $true;

Register-ObjectEvent -InputObject $process -EventName "OutputDataReceived" -Action { Add-Content 'c:\jenkins\log.txt' $args[1].Data };  
Register-ObjectEvent -InputObject $process -EventName "ErrorDataReceived" -Action { Add-Content 'c:\jenkins\log.txt' $args[1].Data };

$process.StartInfo;
$process.Start();
$process.BeginOutputReadLine()
$process.BeginErrorReadLine()

#& $javaExe -jar $slaveJarFilePath -jnlpCredentials $credentials -jnlpUrl $jnlpUrl;

Write-Host 'Done Init Script.';  

Script is also shared as a GitHub Gist.