Human-Readable Ansible Playbook Log Output Using Callback Plugin

One problem I’ve had with Ansible playbook since its early 0.x days is with its verbose log output. Jsonified by default, it’s hard to read, and pretty much impossible for a human to review when its stdout or stderr contains tens/hundreds of lines combined into one lengthy string.

Here’s how it looks like:

changed: [gennou.local] => {"changed": true, "cmd": "/tmp/sample.sh",
"delta": "0:00:00.019164", "end": "2014-03-30 21:05:33.994066", "rc": 0,
"start": "2014-03-30 21:05:33.974902", "stderr": "", "stdout": "gazillion
texts here with lots of \n in between gazillion texts here with
lots of \n in between gazillion texts here with lots of \n
in between gazillion texts here with lots \n in between"}

When –verbose flag is set, I believe that the intention is for a human to eventually review the verbose log output. And whenever the human did review the log, the person never failed to tell me that the jsonified message was impossible to read, to which I replied with “They will fix it someday.”

Well, Ansible is now at version 1.x and the problem is still there.

So, while we continue on waiting, the workaround I use for now is to set up an Ansible callback plugin that listens to some task events, and then logs the result in a human readable format, with each field on its own line and newline stays as-is.

Here’s how I set it up:

  1. Set callback plugins directory in Ansible configuration file (ansible.cfg file):
    [defaults]
    callback_plugins = path/to/callback_plugins/
  2. Create a callback plugin file at path/to/callback_plugins/ directory, I call mine human_log.py .
    Here’s the callback plugin gist: https://gist.github.com/cliffano/9868180
  3. Run ansible-playbook command:
    ansible-playbook -i hosts playbook.yml

And the log output looks like this:

cmd:
/tmp/sample.sh

start:
2014-03-30 21:05:33.974902

end:
2014-03-30 21:05:33.994066

delta:
0:00:00.019164

stdout:
gazillion texts here with lots of
in between
gazillion texts here with lots of
in between
gazillion texts here with lots of
in between
gazillion texts here with lots of
in between
stderr:

Now that’s more readable.

You can set the callback plugin on each Ansible project if you want to. But I set mine as part of my CI/Jenkins boxes provisioning, that way all Jenkins jobs that execute an Ansible playbook end up with a readable log output.

Note: I know that some people suggest using debug module to split output into multiple lines. However, having to add register and debug fields all over the tasks would easily clutter the playbook. I find the callback plugin to be a cleaner and simpler solution.

Jenkins Build Slaves On A Budget

About half a year ago our team started working on a project with micro-service architecture, which means we had a lot of little applications to build as part of our delivery pipeline. One of the reasons why we opted to use this architecture was to gain the ability to replace a piece of component without having to rebuild the whole system, hence enabling faster feedback loop by releasing small chunks of changes in small parts of the system.

But there was one problem. Each application build was CPU-intensive, this includes fetching source, installing dependencies, unit testing, code coverage, integration testing, acceptance testing, packaging, publishing to repositories, and deploying to target environments. And nope, we don’t have a build farm!

Our goal was to have each application build to finish in less than a minute, which was fine when there’s only one build, but failed horribly when there were lots of changes triggering multiple downstream builds, often up to 15 builds at the same time with only 4 executors (4 CPUs) available on the build master. Scaling up wasn’t going to take us far, so we had to scale out and distribute the build system earlier than we normally would with past monolithic stonehenge projects.

We considered the idea of using the cloud, either an existing cloud CI solution or Amazon EC2, but we had to rule this out at the end due to extra cost, source code restriction, and network latency. One developer then suggested the idea of using the developer machines as build slaves, each one having 8 CPUs, SSD, lots of RAM and disk space, plenty of firepower lying around under-utilised most of the time.

So we gave it a go and it worked out really well. We ended up with additional 7×4 = 28 build executors, and it’s not unusual to have those 15 applications built at the same time and finished within a minute. Here’s our setup:

  • Each build slave has to self-register to the master because developer machines only have dynamic IP addresses, so they can’t be pre-configured on build master.
    This is where Jenkins Swarm Plugin comes in handy, allowing each slave to join the master, and the master doesn’t need to know any of the slaves beforehand.
  • Each build slave has to re-register when the machine is rebooted, we use upstart to do this.
  • Each build slave runs as its own user, separate from the developer’s user account. This allows a clean separation between user workspaces.
  • Each build slave is provisioned using Ansible, always handy when there are more build slaves to add in the future, or to update multiple build slaves in one go.
  • Each build slave is allocated 50% of the available CPUs on the machine to reduce any possibility of interrupting developer’s work.

So there, build slaves on a budget :).

I think it’s easy to overlook the fact that developer/tester machines are often under utilised, and that they would serve an additional purpose as Jenkins build slaves, a reasonable alternative before looking at other costlier solutions.

CITCON 2013

I attended CITCON 2013 in Sydney last February. This year’s sessions covered more non-technical issues compared to CITCON 2010. Two of the more interesting topics for me were on how devops movement could potentially discourage collaboration, and on how large non-tech companies try and still fail to implement continuous delivery.

Those were some of the problems that I’ve been battling for many years. In an organisation where dev and ops are two separate divisions, devops is often a shortcut for dev to do ops tasks while bypassing any ops involvement. Instead, a better alternative would be for dev and ops teams to collaborate and stop fighting over issues like root access.

As for the second topic, continuous delivery is sometimes not as straightforward as it seems. One major obstacle to continuous delivery implementation is a conservative change management process. No matter how you automate your delivery pipeline along your development, test, and staging environments, it would all be useless if production deployment requires manual approval for the sake of auditing.

Technology is often the easier part, the harder part is on people and policies, on changing a culture, on accepting new ideas.

The best part of CITCON has always been its open space format where ideas/opinions/experiences flow during the discussions. And like most tech conferences, the hallway discussions were not to be missed. The quote-of-the-conf went to Jeffrey Fredrick for pointing out that (my interpretation of what he said) technologists often suck for focusing on the technology to sell to the business (e.g. continuous delivery is awesome), instead of focusing on the business problem and how the technology can solve it (e.g. business problem is time to market, continuous delivery can help).

I also caught up with Michael Neale from CloudBees there, here’s his CITCON 2013 notes, along with some familiar faces from CITCON 2010.

How To Incorrectly Track The Progress Of A Project

The next time a manager asks you the percentage progress of a task, make sure you reply with a float rounded to 2 decimal places, and say it with full confidence.

Sometime in mid 2000s, the manager of a project I was working on had to take an emergency leave, so another manager filled in for him. It was an agile project, but somehow this other manager managed to come up with a magical Gantt chart for the rest of the project. The project was waterfalled!

The task in question was titled “Create HTML”, a bucket task to dump all UI related tasks from the look of it. Unfortunately I didn’t have the guts to answer with 83.62% back then, and I replied with a magical whole number divisible by 10 instead.

I know that there are still people believing in waterfall methodology to this day (yes, in 2013, believe it or not), and if it works for them, good on them. But whatever the methodology is, asking for a percentage is not going to give you the right idea of how the project is actually progressing.

It makes a good practice for generating magic numbers in your head though :).

Private NPM Registry Replicator Document

For my future reference and to help others trying to set up a private NPM registry which sits behind a [corporate] proxy and requires authenticated CouchDB admin access, here’s the replicator document that I ended up using:

{
   "_id": "registry",
   "source": "http://isaacs.iriscouch.com/registry/",
   "target": "registry",
   "user_ctx": {
       "name": "myadmin_username",
       "roles": ["_admin"]
   },
   "continuous": true,
   "owner": "myadmin_username",
   "proxy": "http://proxy:8080"
}

For those who are not familiar with CouchDB, the above is a document that needs to be created in _replicator database, so that the replication rule from public NPM registry to your private NPM registry can be persisted and runs continuously.

The key here is the user_ctx, owner, continuous, and proxy settings, which after several trials and errors, I managed to get them right with several clean full replication runs (always monitor CouchDB log during replication!).

Have a look at CouchDB Replication wiki page and this gist on the new replicator database (introduced in CouchDB v1.2.0) for explanation on those fields and how CouchDB replication works.

Tip:

If you tried to be smart by copying the design documents from the public NPM registry to your private NPM registry before replicating the documents, you would see some errors in CouchDB log file due to some rules in the latest design document that would fail the older documents. For example: module name must be in lower case, but there are old modules with name containing upper case letter(s).

My suggestion is to start from empty registry and public_users databases, then kick off the replication, and refresh the indices nearing the end of the replication, followed by compacting the databases and views to save some disk space.

Tip 2:

Public NPM registry sets require_valid_user = false in its CouchDB configuration file, which allows database read access without CouchDB admin authentication. If you set require_valid_user = true instead, then CouchDB will require authentication when you fetch any module (a document from CouchDB’s point of view). Unfortunately, as of NPM v1.2.15, there’s no authentication info on its fetch request, so you might find this monkey patch and this issue handy.

Tip 3:

And if you want to make the private NPM registry available over SSL:

  1. Generate self-signed SSL certificate
  2. Enable SSL on CouchDB via local.ini configurations
  3. Configure registry URL in .npmrc file to use https://
  4. npm install

If you get DEPTH_ZERO_SELF_SIGNED_CERT error, you might want to check this issue for workarounds.
If you’re connecting via a proxy, you might get ‘tunneling socket could not be established, cause=Parse Error’ error.

Tip 4:

If you are working behind an older proxy server, there’s a chance that your replication might fail because the proxy rejects lengthy GET request URL. To get around this problem, you need to patch MAX_URL_LEN with a larger value. /hattip: Adam Kocoloski

Tip 5:

If you are seeing this error:

[error] [<0.18564.6>] Uncaught server error: {insecure_rewrite_rule, <<“too many ../.. segments”>>}

You need to set this in CouchDB configuration .ini file:

[httpd]
secure_rewrites=false

Sapi, A Node.js Client For Sensis API

We had a hack day at Sensis a couple of days ago where I ended up writing Sapi, a Node.js client for Sensis API. This module is now available on NPM.

The latest version (v0.0.5) was tested against Sensis API open beta version ob-20110511. I think I’ve got all of its documented features covered. Let me know if I missed anything.

Sapi module provides a chainable interface to construct the endpoint parameters, here’s an example:

sapi
  .query('restaurants')
  .location('Melbourne')
  .search(function (err, result) {
    ...
  });

The above snippet performs a search for restaurants in Melbourne. It uses Sensis API search endpoint with query and location parameters.

If you want to add more parameters, e.g. for postcode and state filtering, just keep chaining them like this:

sapi
  .query('restaurants')
  .location('Melbourne')
  .postcode('3000')
  .state('VIC')
  .search(function (err, result) {
    ...
  });

Simple enough? Check out Sapi README for installation and further usage guide.

So, if you would like to write an application which requires Australian business listings information, check out Sensis API and apply for a key. And if you’re a Node.js coder, give Sapi a try.

Note: I know there’s also node-sapi module published to NPM, but it seems to be just a placeholder as the source code currently only implements API key checking.

Publishing Node.js Module To Ivy Repository

Let me guess what you’re going to say in 3… 2… 1…

WTF??? Why would anyone want to do that?

Right?

Some of us don’t have the luxury of a local NPM repository, while some others have their delivery pipeline tightly integrated to an Ivy repository.

So, for those few who are stuck with the unholy union of Node.js and Apache Ivy, you can publish your Node.js module to an Ivy repository using Bob. Here’s how:

Update (23/08/2012): The instruction below is for Bob v0.4.x or older. If you’re using Bob v0.5.0, please scroll further down for the updated instruction.

1. Create an ivy.xml file template in your Node.js module project directory

<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="com.company" module="modulename" status="integration" revision="${version}" publication="${now('yyyymmddHHMMss')}"/>
    <configurations>
        <conf name="default" visibility="public" description="..." extends="runtime,master"/>
    </configurations>
    <publications>
        <artifact name="modulename" type="tar.gz" conf="default"/>
    </publications>
</ivy-module>

2. Create a .bob.json file in your project directory, specifying ivy.xml location and details of the Ivy repository server

{
  "packagemeta": {
    "dir": "path/to",
    "file": "ivy.xml"
  },
  "template": [
    "path/to/ivy.xml"
  ],
  "deploy": {
    "user": "username",
    "key": "path/to/keyfile",
    "host": "hostname",
    "port": portnumber,
    "dir": "/path/to/ivy/repo/${name}/${version}"
  }
}

3. Run Bob

bob template package package-meta ssh-mkdir deploy (yup, this could’ve been simpler, on my TODO list)

This will create /path/to/ivy/repo/modulename/version/ directory with the following files:

  • modulename.tar.gz
  • modulename.tar.gz.md5
  • modulename.tar.gz.sha1
  • ivy.xml
  • ivy.xml.md5
  • ivy.xml.sha1

This module can then be referenced as com.company.modulename, and used just like any other artifact using Ivy.

If the repository is accessible via HTTP, then you can also specify the Ivy artifact as a dependency of another Node.js module in its package.json file:

{
  "dependencies": {
    "modulename": "http://ivyserver/com/company/modulename/version/modulename-version.tar.gz",
  }
}

Update (23/08/2012): The instruction below is for Bob v0.5.x or newer.

1. Create an ivy.xml file template in your Node.js module project’s root directory
If you’re upgrading from Bob v0.4.x, all you need to do is remove the $ from the parameter syntax.

<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ant.apache.org/ivy/schemas/ivy.xsd">
    <info organisation="com.company" module="modulename" status="integration" revision="{version}" publication="{now('yyyymmddHHMMss')}"/>
    <configurations>
        <conf name="default" visibility="public" description="..." extends="runtime,master"/>
    </configurations>
    <publications>
        <artifact name="modulename" type="tar.gz" conf="default"/>
    </publications>
</ivy-module>

2. Create a .bob.json file in your project directory, specifying ivy.xml location and details of the Ivy repository server
If you’re upgrading from Bob v0.4.x, you need to move ivy.xml to the project’s root directory, and modify .bob.json by removing packagemeta, change template structure, renaming deploy to publish, adding publish.type: ivy .

{
  "template": [".bob/artifact/ivy.xml"],
  "publish": {
    "type": "ivy",
    "user": "username",
    "key": "path/to/keyfile",
    "host": "hostname",
    "port": portnumber,
    "dir": "/path/to/ivy/repo/${name}/${version}"
  }
}

3. Run Bob
If you’re upgrading from Bob v0.4.x, simply replace template package package-meta ssh-mkdir deploy targets, with package publish

bob package publish

This will create /path/to/ivy/repo/modulename/version/ directory with the following files:

  • modulename.tar.gz
  • modulename.tar.gz.md5
  • modulename.tar.gz.sha1
  • ivy.xml
  • ivy.xml.md5
  • ivy.xml.sha1

This module can then be referenced as com.company.modulename, and used just like any other artifact using Ivy.

If the repository is accessible via HTTP, then you can also specify the Ivy artifact as a dependency of another Node.js module in its package.json file:

{
  "dependencies": {
    "modulename": "http://ivyserver/com/company/modulename/version/modulename-version.tar.gz",
  }
}

Despite my dislike towards XML configuration files, Ivy has worked just fine all these years and I’ve been using it to store various types of artifacts. Even though its main popularity is within the Java community, you can store pretty much anything there (YMMV).

Bob itself is still at an early stage, there are lots of things I want to improve. I just need that mythical spare time :).