Wrapping AEM cURL Commands With Python

If you ever had the experience (no pun) of using Adobe Experience Manager (AEM), you would already know that curl commands are arguably the de facto way of interacting with AEM over http.

Whenever you google for various AEM /CQ HOWTOs, it’s easy to find examples with curl commands:

Naturally I started integrating those curl commands into my project’s application provisioning and deployment automation via Ansible’s shell module. However, it wasn’t long until I encountered a number of issues:

  • Lack of consistent response payload format from AEM. Some status messages are embedded within various html response bodies, some within json objects.

  • Some endpoints are returning status code 500 for non-server error result (e.g. when an item to be created already exists), making it hard to differentiate from a real server error.

  • Some endpoints are returning status code 200 with error message in the html response body.

  • Even though curl –fail exists, it’s not fail-safe. There doesn’t seem to be any way to identify success/failure result without parsing the response headers and body.

  • Which means that curl commands could be returning exit code 0 even when the http status code indicates an error, and Ansible would not fail the task, it would simply continue on to the next task.

  • Printing the response bodies to stdout won’t help much either, it will be painful for a human having to go through a large volume of text to identify any error.

It’s obvious that curl commands alone are not enough, I need better error handling by both checking status code and parsing response body, and then translating it into Ansible success/failed status. So I wrote PyAEM PyAEM, a Python client for Adobe Experience Manager (AEM) API.

Why Python? 1) It’s first class in Ansible. 2) It’s saner to handle the response (status code checking, html/json parsing) in Python compared to shell. 3) Ditto for code lint, unit tests, coverage check, and package distribution, Python wins!

PyAEM ended up using pycurl to simplify porting those curl commands into Python. I initially tried Requests instead and managed to port majority of the curl commands, until I got to package manager API and  kept getting different responses from AEM with Requests compared to the ones with curl commands. Since AEM was a black box and I didn’t have any access to its source code, I couldn’t tell what was it with libcurl and package upload/download that was missing from Requests. So at the end I stuck with pycurl.

Here’s a code snippet on how to use PyAEM to stop a bundle called ‘mybundle’: (check out PyAEM API Reference to find out other actions that PyAEM currently supports)

import pyaem

aem = pyaem.PyAem('admin', 'password', 'localhost', 4502)

try:
    result = aem.stop_bundle('mybundle')

    if result.is_success():
        print 'Success: {0}'.format(result.message)
    else:
        print 'Failure: {0}'.format(result.message)
except pyaem.PyAemException, e:
    print e.message

Better. Now it has success/failure status handling and also error handling by catching PyAemException.

As for Ansible, the next obvious step is to create Ansible modules which utilise PyAEM. These modules serve as a thin layer between Ansible and PyAEM, all they need to worry about is argument passing and status handling.

#!/usr/bin/python

import os
import pyaem
import commands

def main ():
    module = AnsibleModule(
        argument_spec = dict(
            host = dict(required=True),
            port = dict(required=True),
            bundle_name = dict(required=True)
        )
    )

    host = module.params['host']
    port = module.params['port']
    bundle_name = module.params['bundle_name']

    aem_username = os.getenv('crx_username')
    aem_password = os.getenv('crx_password')

    aem = pyaem.PyAem(aem_username, aem_password, host, port)
    result = aem.stop_bundle(bundle_name)

    if result.is_failure():
        print json.dumps({ 'failed': True, 'msg': result.message })
    else:
        print json.dumps({ 'msg': result.message })

from ansible.module_utils.basic import *
main()

The above module can then be used in an Ansible playbook.

- name: 'Stop com.day.crx.crxde-support bundle'
  aem-stop-bundle: >
    host=somehost.com
    port=4503
    bundle_name=com.day.crx.crxde-support

Too simple!

This can actually be improved further by creating an Ansible role for AEM, distributed through Galaxy. Things like downloading an AEM package file from an artifact repository, uploading the package to AEM, install, then replicate, it’s a repetitive pattern for AEM package management.

PyAEM is still at an early stage, but it’s stable enough (we use it in production). It currently only supports the actions that are used in my project. Having said that, I think the package is pretty solid with 100% unit test coverage, 0 lint violation, and an automated Travis CI build on every code change.

Since AEM is a proprietary product, it currently doesn’t have any automated integration tests (think AEM docker containers :) ). However, PyAEM is verified to work with AEM 5.6.1 and Python 2.6.x/2.7.x via the internal project I’m working on.

Want to use it? PyAEM is available on PyPI. Anything missing? Contributions are welcome!

Update (30/12/2014):

PyAEM is no longer controlled by me starting from October 2014, the source code was forked to this repository and I already deleted the original repo under my GitHub account as requested. I cannot comment on the future of PyAEM, but if you’re interested, I can find the information for you (ping me at blah@cliffano.com).

If anyone is interested in writing a PyAEM-like library from scratch, I can convince you that it’s not hard but it can be time consuming. Various AEM API documentations are available in public so there’s nothing that can’t be added to the library. I’m happy to help if you have any question.

Lastly, thanks to the folks who already starred the original repo and to those who took the time to tell me that PyAEM was useful. Your feedback is much appreciated!

Share Comments
comments powered by Disqus