CouchDB Benchmarks: gcc vs. clang

Since the release of Snow Leopard, I’ve been itching to run some benchmarks. Jan Lehnhardt suggested I give the benchmarking script from this bug a few runs. So that’s what I did.

At first glance testing seemed pretty simple. Compile one version of Erlang with gcc and another with clang. Then use each of those versions to compile CouchDB and run benchmarks against each Erlang/CouchDB pair. Since CouchDB also has at least one bit of native code, couchjs, I’d also used gcc and clang to compile that, too.

Then reality showed up and ruined my day. Compiling CouchDB on OS X is, well, a pain in the ass involving MacPorts and lots of dependencies if you do it wrong. Doing it right means grabbing Jan’s couchdbx-core project from Github and building new guts for CouchDBX. This is the way to go, IMHO. 10 – 15 minutes’ of compiling gave me a new Erlang, SpiderMonkey, and CouchDB I could drop into the Contents/Resources/couchdbx-core directory of my previously installed CouchDBX app.

Compiling with clang is similarly easy. All you need to do is set the environment variable CC to point to /Developer/usr/bin/clang. I found I also needed to set CFLAGS to -Qunused-arguments to get Erlang to compile.

The benchmark itself consists of three phases. The first phase creates a database and populates it with 40,000 documents. The second phase creates and executes a complex view named 'megaview' to really exercise the view collation code. The last phase creates and executes a simple view named 'simpleview'.

Let me quickly describe how the performance numbers were collected. The benchmark was executed 5 times against the clang and gcc versions of CouchDB. I've computed a raw average across all runs as well as a "normalized" average which throws out the highest and lowest times. Finally, I computed the standard deviation as a indicator of overall variability.

Enough talk. Here are the numbers:

Note: Document insertion times are given in seconds while view collation numbers are given in minutes

The megaview test really highlights the disparity between clang and gcc. It's worth noting the test also showcases a bottleneck in CouchDB and only used a single core on my laptop. If I had to guess I'd say the performance disparity is a function of the age of each compiler. gcc has been around forever so it makes sense it's had more time to implement more/better optimizations than clang.