Private NPM Registry Replicator Document

For my future reference and to help others trying to set up a private NPM registry which sits behind a [corporate] proxy and requires authenticated CouchDB admin access, here’s the replicator document that I ended up using:

   "_id": "registry",
   "source": "",
   "target": "registry",
   "user_ctx": {
       "name": "myadmin_username",
       "roles": ["_admin"]
   "continuous": true,
   "owner": "myadmin_username",
   "proxy": "http://proxy:8080"

For those who are not familiar with CouchDB, the above is a document that needs to be created in _replicator database, so that the replication rule from public NPM registry to your private NPM registry can be persisted and runs continuously.

The key here is the user_ctx, owner, continuous, and proxy settings, which after several trials and errors, I managed to get them right with several clean full replication runs (always monitor CouchDB log during replication!).

Have a look at CouchDB Replication wiki page and this gist on the new replicator database (introduced in CouchDB v1.2.0) for explanation on those fields and how CouchDB replication works.


If you tried to be smart by copying the design documents from the public NPM registry to your private NPM registry before replicating the documents, you would see some errors in CouchDB log file due to some rules in the latest design document that would fail the older documents. For example: module name must be in lower case, but there are old modules with name containing upper case letter(s).

My suggestion is to start from empty registry and public_users databases, then kick off the replication, and refresh the indices nearing the end of the replication, followed by compacting the databases and views to save some disk space.

Tip 2:

Public NPM registry sets require_valid_user = false in its CouchDB configuration file, which allows database read access without CouchDB admin authentication. If you set require_valid_user = true instead, then CouchDB will require authentication when you fetch any module (a document from CouchDB’s point of view). Unfortunately, as of NPM v1.2.15, there’s no authentication info on its fetch request, so you might find this monkey patch and this issue handy.

Tip 3:

And if you want to make the private NPM registry available over SSL:

  1. Generate self-signed SSL certificate
  2. Enable SSL on CouchDB via local.ini configurations
  3. Configure registry URL in .npmrc file to use https://
  4. npm install

If you get DEPTH_ZERO_SELF_SIGNED_CERT error, you might want to check this issue for workarounds.
If you’re connecting via a proxy, you might get ‘tunneling socket could not be established, cause=Parse Error’ error.

Tip 4:

If you are working behind an older proxy server, there’s a chance that your replication might fail because the proxy rejects lengthy GET request URL. To get around this problem, you need to patch MAX_URL_LEN with a larger value. /hattip: Adam Kocoloski

Tip 5:

If you are seeing this error:

[error] [<0.18564.6>] Uncaught server error: {insecure_rewrite_rule, <<“too many ../.. segments”>>}

You need to set this in CouchDB configuration .ini file:


Node.js Presentations

I gave two Node.js-related talks within the past week.

The first one was titled “From Java To Node.js”, at Shine Technologies‘ developers meeting on August 5th, 2011.

The second one was titled “JavaScript Everywhere From Nose To Tail”, at Melbourne JavaScript usergroup on August 10th, 2011, with Carl Husselbee from Sensis.

Happy with the positive feedback from the audience of both talks, thanks folks, much appreciated!

Update (08/09/2011):

And here’s the video from the second talk…

JavaScript Everywhere – From Nose To Tail from Benjamin Pearson on Vimeo.

Running CouchDB-Lucene On Tomcat

As of the time of writing, CouchDB-Lucene (v0.6.0) distribution was bundled with Jetty WebServer and runnable from command line. But if you dive a little bit into the code, you’ll find that the run script basically executes com.github.rnewson.couchdb.lucene.Main, which in turn (1) reads the couchdb-lucene.ini config, (2) creates a Jetty Server, and (3) sets up com.github.rnewson.couchdb.lucene.LuceneServlet on the server.

At work, we had to use Tomcat due to SOE reason, so I spent a bit of time trying to get CouchDB-Lucene running on Tomcat and get it deployed as a .war file. Here’s how we did it, please note that this is the tedious way of doing it, all because I had to integrate CouchDB-Lucene into an existing web app. There’s an easier way to build a .war file mentioned at the end of this post.

  1. Unpack couchdb-lucene-<version>.zip .
  2. Create a standard web app directory structure with WEB-INF dir.
  3. Configure LuceneServlet in your-app/WEB-INF/web.xml .
  4. Copy all jars from couchdb-lucene-<version>/lib/ to your-app/WEB-INF/lib/ . The jetty-*.jar files can be excluded.
  5. Copy couchdb-lucene-<version>/conf/couchdb-lucene.ini and couchdb-lucene-<version>/conf/log4j.xml to your-app/WEB-INF/classes/ .
  6. Deploy your-app on Tomcat.
  7. Configure the CouchDB external process hook script with the host name and port number where Tomcat is listening on.

There was one catch though, LuceneServlet did not have any no-argument constructor for the web.xml configuration, so that’s one of the changes I had to make. The other change was some file reorganisation so that you can use mvn war:war to generate a target/couchdb-lucene-<version>.war file. Too easy!

I’ll submit a pull request to master later, but in the mean time you can clone my fork to get the patched LuceneServlet and to easily generate the .war file.

Update (27/05/2011): It’s now part of couchdb-lucene master as of 0081272a30dc679effc1cf1298e365b953f568a5, should be included in v0.7.0 .

CouchDB Upgrade From 0.11.0 To 1.0.1

This post also serves as a note to self when I do another upgrade in the future.

It’s still not clear to me of what’s the best practice for upgrading a CouchDB installation. From the mailing list, some people suggested backing up the config and database files, uninstalling the old version, then installing the new version. But, really, what’s involved in uninstalling a CouchDB installation? manually deleting the files from the old installation? I had the same confusion when I tried to do the upgrade on OS X, once from a compile from source, and another from a CouchDBX installation.

So instead of ‘the best practice’, I’ll just share what worked for me. I simply installed the new version using the same way as when I installed the old version, i.e. compile from source, and the files from 1.0.1 installation overwrote the files from 0.11.0 . Here’s what I did:

  1. I searched for CouchDB config and database files, and found them in /usr/local/etc/couchdb/local.ini and /usr/local/var/lib/couchdb/mydatabase.couch . I then backed up those files.
  2. The fact that I found those files under /usr/local indicated that I must’ve used –prefix=/usr/local when I installed 0.11.0 .
  3. I downloaded the source code of CouchDB 1.0.1, then ran the usual:
    ./configure --prefix=/usr/local
    sudo make install
  4. Voila, CouchDB 1.0.1 was installed, the config and database files were still intact, everything worked fine.