Monitor Jenkins From The Terminal

Here’s how I’ve been monitoring my Jenkins setup…

A combination of Nestor + watch + Terminator » one view for monitoring failing builds, one view for executors status, and one view for job queue. A summary of Jenkins status info on a small screen estate that I can place at the corner of my workspace.

If you want to set up something similar, here are the commands: (assume JENKINS_URL is already set)

  • watch -c “nestor dashboard | grep FAIL”
  • watch nestor executor
  • watch nestor queue

Jenkins Build Slaves On A Budget

About half a year ago our team started working on a project with micro-service architecture, which means we had a lot of little applications to build as part of our delivery pipeline. One of the reasons why we opted to use this architecture was to gain the ability to replace a piece of component without having to rebuild the whole system, hence enabling faster feedback loop by releasing small chunks of changes in small parts of the system.

But there was one problem. Each application build was CPU-intensive, this includes fetching source, installing dependencies, unit testing, code coverage, integration testing, acceptance testing, packaging, publishing to repositories, and deploying to target environments. And nope, we don’t have a build farm!

Our goal was to have each application build to finish in less than a minute, which was fine when there’s only one build, but failed horribly when there were lots of changes triggering multiple downstream builds, often up to 15 builds at the same time with only 4 executors (4 CPUs) available on the build master. Scaling up wasn’t going to take us far, so we had to scale out and distribute the build system earlier than we normally would with past monolithic stonehenge projects.

We considered the idea of using the cloud, either an existing cloud CI solution or Amazon EC2, but we had to rule this out at the end due to extra cost, source code restriction, and network latency. One developer then suggested the idea of using the developer machines as build slaves, each one having 8 CPUs, SSD, lots of RAM and disk space, plenty of firepower lying around under-utilised most of the time.

So we gave it a go and it worked out really well. We ended up with additional 7×4 = 28 build executors, and it’s not unusual to have those 15 applications built at the same time and finished within a minute. Here’s our setup:

  • Each build slave has to self-register to the master because developer machines only have dynamic IP addresses, so they can’t be pre-configured on build master.
    This is where Jenkins Swarm Plugin comes in handy, allowing each slave to join the master, and the master doesn’t need to know any of the slaves beforehand.
  • Each build slave has to re-register when the machine is rebooted, we use upstart to do this.
  • Each build slave runs as its own user, separate from the developer’s user account. This allows a clean separation between user workspaces.
  • Each build slave is provisioned using Ansible, always handy when there are more build slaves to add in the future, or to update multiple build slaves in one go.
  • Each build slave is allocated 50% of the available CPUs on the machine to reduce any possibility of interrupting developer’s work.

So there, build slaves on a budget :).

I think it’s easy to overlook the fact that developer/tester machines are often under utilised, and that they would serve an additional purpose as Jenkins build slaves, a reasonable alternative before looking at other costlier solutions.

Jenkins Build Status On Ninja Blocks RGB LED

Nestor v0.1.2 is out and one of its new features is nestor ninja for monitoring Jenkins and displaying the latest build status on Ninja Blocks RGB LED device (if you have a block, it’s the ninja’s eyes).

Here’s a usage example:
export JENKINS_URL=<url>
export NINJABLOCKS_TOKEN=<token_from_https://a.ninja.is/hacking>
nestor ninja

Red for build failure, green for build success, yellow for build warning, and white for unknown status. The yellow light looks quite similar to green, and the white one does look blue-ish.

And the best place to run nestor ninja? On the block itself of course!

ssh ubuntu@ninjablock.local
apt-get install upstart
npm install -g nestor
cat /usr/lib/node_modules/nestor/conf/ninja_upstart.conf > /etc/init/nestor_ninja.conf
vi /etc/init/nestor_ninja.conf # and change JENKINS_URL and NINJABLOCKS_TOKEN values
shutdown -r now

Log messages will then be written to /var/log/nestor_ninja.log

OSDC 2011

I went to Canberra this week to attend Open Source Developers Conference 2011 and also to give a talk titled Continuous Delivery Using Jenkins. OSDC ran for 3 days, and was held at Australian National University.

OSDC 2011 was very well organised, much thanks to the organisers: Evan Leybourn, Gavin Jackson, and the volunteers squad. It was an interesting grass roots conference with lots of passionate open source geeks, definitely learned a lot.

Slides from my talk:

Update (24/11/2011): and the video of the talk:

Use Jenkins On Firefox By Talking To Your Mac

I was watching I, Robot the other day, and thought how great it would be to use voice to control Jenkins.

So last night I did a quick read, and then recorded this video.
This used Mac Speech Recognition, so in theory I talked to my Mac, which then opened Jenkins pages on Firefox. Here are the commands:

  1. ‘Open Jenkins': opens Jenkins home page (dashboard)
  2. ‘Build Bob': builds a project called Bob
  3. ‘Configure system': opens Jenkins configuration page

I had to repeat each command 2-3 times because I speak Indonesian-accented English.

Note: I scrolled the page up and down using the trackpad, it could be voice-controlled too actually.

It’s easy to add a command:

  1. Create a new file in /Users/<username>/Library/Speech/Speakable Items/Application Speakable Items/firefox directory containing:
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
    <key>URL</key>
    <string>http://jenkins-host:8080</string>
    </dict>
    </plist>
    

    This format is Mac OS X Property List.

  2. Save this file as <command>, e.g. Open Jenkins (yes, with the space).
  3. Open System Preferences -> Speech, and switch Speakable Items on.
  4. Configure the Listening Method. I set it to Listen continuously with keyword, and keyword is Required before each command. My keyword is VIKI, you know, Virtual Interactive Kinetic Intelligence, from the movie.

I haven’t investigated how much programming can be done on the command file, but this opens up the possibility of mapping Jenkins HTTP API to commands, and we will be able to fully interact with Jenkins using voice.

Then the only thing left to do is… develop an American accent.

Nestor – A Faster And Simpler CLI For Jenkins

It all started because at one point I was using a rather resource-challenged machine running Windows and an Ubuntu VM at the same time, and Firefox froze every so often, rendering Jenkins BuildMonitor and Jenkins web interface useless most of the time. So I looked for an alternative and gave Jenkins CLI a go.

Like most Java applications, Jenkins built-in CLI also suffers from slow start up time (flame suit: ON) due to core Java libraries loading (Kohsuke later told me on #jenkins that there’s also a handshaking process involved). This led me to try Jenkins Remote Access API with curl, which performed significantly faster than Jenkins CLI.

So that’s great, but I have another issue with the fact that Jenkins CLI’s commands start with “java -jar jenkins-cli.jar …”, that’s a finger twister right there, and lengthy curl + URL obviously doesn’t help.

Enter Nestor, a Jenkins CLI written in Node.js that aims to be a faster and simpler alternative to the existing solutions. The catch? Node.js and npm support on Windows is not there yet, so if you managed to run Nestor on Windows please let me know about it. Nestor has been tested and used daily on OS X and Linux.

Simple setup

Install Nestor using npm install -g nestor

Configure the Jenkins instance you want to use using export JENKINS_URL=http://user:pass@host:port/path

Simple usage

Nestor commands are simple, it’s always nestor <action> <param>

To trigger a build

> nestor build studio-bob
Job was started successfully

To view a job status

> nestor job studio-bob
Status: OK
No xml report files found for checkstyle
Build stability: 3 out of the last 5 builds failed.

To list the executors

> nestor executor
* master
idle
39%	studio-bob

To view the queue

> nestor queue
Queue is empty

To view all jobs status on the dashboard

> nestor dashboard
WARN	blojsom-bloojm
OK	jenkins-buildmonitor
FAIL	studio-ae86
OK	studio-bob

Check out Nestor’s GitHub README page for more commands available.

Hopefully that’s simple enough.

Note: The name Nestor was inspired by Captain Haddock’s butler at Marlinspike Hall, not the Argonaut one.

Using Node.js To Discover Jenkins On The Network

I’ve just added a new feature to Nestor to discover Jenkins on the network, and as it turned out, it’s pretty simple to do thanks to Node.js Datagram sockets API (hat tip Paul Querna).

Jenkins has a discovery feature as part of its remote access API where it listens on UDP port 33848, and whenever it receives a message, Jenkins will respond with an XML containing the instance’s URL, version number, and slave port information.

So how do you send a UDP message using NodeJS?
Here’s a sample function adapted from Nestor’s lib/service.js:

function sendUdp(message, host, port, cb) {
    var socket = require('dgram').createSocket('udp4'),
        buffer = new Buffer(message);
    socket.on("error", function (err) {
        cb(err);
    });
    socket.on("message", function (data) {
        cb(null, data);
    });
    socket.send(buffer, 0, buffer.length, port, host, function (err, message) {
        if (err) {
            cb(err);
        }
    });
}

For Jenkins discovery purpose, send any message to any hostname on port 33848:

sendUdp('Long live Jenkins!', 'localhost', 33848, function () { ... });

and if there’s any Jenkins instance running on localhost, it will respond with an XML like this:

<hudson>
  <version>1.414</version>
  <url>http://localhost:8080</url>
  <slave-port>12345</slave-port>
</hudson>

Simple!

konan cliffano$ nestor discover
Jenkins 1.414 running at http://localhost:8080/

Jenkins BuildMonitor – 20000 Downloads Later

Yesterday, Jenkins BuildMonitor Firefox Add-on surpassed 20,000 downloads. I uploaded the first version on June 8th, 2008, so that’s almost 3 years ago, and there have been 26 releases since.

On average, there are about 2500-2700 active daily users on weekdays. That became the basis of my release mantra: “Let’s not piss 2500 people off.”

The top locales are en-US, de, en-GB, fr, ja, pl. We’ve got en, fr, and ja locales covered https://github.com/jenkinsci/firefox-extension-buildmonitor/tree/master/src/main/resources/firefox/chrome/locale. Any German or Polish translator around? please feel free to submit a pull request.

While the top operating systems are Windows, Linux, Darwin, Solaris, FreeBSD.

And as you know, this add-on is now named Jenkins BuildMonitor, though in practice it should still work with both Jenkins and Hudson just fine. As for the project management, I’m staying with the original Hudson – now Jenkins – community, source code on GitHub, mailing list on Google Groups, etc.