Hello, Bob. Hello Joe.

MochiAds launch has really intrigued me. How does a modern technology company use Erlang? How hard is it to find Erlang-savvy talent. I was intrigued enough to fire off a few questions to Bob Ippolito, CTO at MochiAds, and get his take on things. Thanks to Bob’s quick and helpful answers, you too can know the answers to these burning questions:

I’ve been programming in Erlang for about a year now, give or take a few weeks.

I first tried Erlang when I was planning for MochiAds. I knew had to
be able to handle lots of TCP traffic and I didn’t have anyone at the
time to help… so I needed multiplexed IO, high-availability, and a
quick develop/test/debug cycle. I also knew I was going to need a
distributed architecture sooner than later so distributed features
were weighted very highly but not an absolute requirement.

I gave myself about two weeks to do the research and I picked out a
couple languages to try out. Erlang was on the short list, and it was
really easy for me to pick up conceptually. The majority of the time I
spent learning Erlang was wasted trying to figure out how to configure
Yaws embedded (which is somewhat easier now, but I still wrote our own
web server), but beyond that it was really quick for me to build a
prototype. When I did some benchmarks on it, it worked so much better
than the Python equivalent that I thought it might actually be broken;
it wasn’t, so Erlang very quickly became my production platform of
choice.

The long story is that I had used Python for all of my other
significant network programming projects and it was either very
arduous (rolling your own network framework), awkward (callback-based
programming a la Twisted), or scary. I say scary because I had
prototyped a networking framework called eventlet using something
called greenlets, an outgrowth of Stackless Python, which gave me
decent performance and it was straightforward to write code with. The
problem was that the debugger no longer behaved properly and I just
didn’t feel safe knowing that my server was squirreling away snippets
of the C stack for task switching around a VM that wasn’t really
expecting that. It apparently works though, because a friend of mine
that works at Linden Lab took the prototype and now they use that in
production and have since open sourced it. At least the effort wasn’t
wasted :)

2. What are you using to write code? Machine, operating system,
editor, etc.

I first started with Erlang on a 1ghz Powerbook G4 running OS X 10.4
and have since upgraded to a pair of 15″ MacBook Pros running the
same. I’m normally a Vim user with a little bit of Textmate, but for
Erlang code I always use Aquamacs Emacs, since erlang mode is just
that good. Not good enough to make me use Emacs for anything else,
though :)

Everyone here that codes in Erlang also uses a MacBook Pro running OS
X 10.4 with the same environment. We have a copy of Erlang, an emacs
config, etc. in a subversion repository which we just check out. It
only takes a few minutes to set up a new dev machine from scratch.

For production we had originally used 32bit x86 FreeBSD machines, but
we started transitioning to 64bit x86 Linux machines. The 64-bit
machines can have a lot more RAM and Linux is a much better supported
platform for Erlang. If someone had fixed that kqueue bug with pipes
then we’d probably be using 64-bit FreeBSD machines.

3. What is your favorite language feature or library?

That’s really hard to answer because there’s a lot I like about
Erlang, but I’d have to say hot-code loading. You can attempt hot-code
loading in Python but it’s too fragile for most practical uses. It
works perfectly in Erlang because the VM and libraries are designed
for it.

If I had to pick a library, it’d be Mnesia. I like Mnesia a lot, but
I’m only really happy with it for ram_copies databases (which is still
extremely useful to us).

4. If you could change one part of the language, what would you change?

Records. Everyone hates records. The syntax sucks, the compile-time
cross-module dependencies they introduce sucks, and they make it hard
to do hot upgrades of systems. I end up using proplists a lot where
most people might use records to get around these annoyances. I’d like
to see something like a syntax supported dict or gb_trees; maybe Joe
Armstrong’s structs proposal.

Adding capability to the language a la E would be cool, but the
security model of Erlang doesn’t get in our way practically. It would
allow different kinds of applications to be built easily, but not the
kind of applications we’re writing at the moment.

5. How/why did you decide to use Erlang for MochiAds?

This is pretty much answered in my response to the first question. I
learned Erlang because I was seeking out a good platform to use for
MochiAds. I prototyped MochiAds in Erlang, and it worked so well that
my prototype became our production server. Erlang code was easy enough
to write and the performance and reliability far exceeded my
expectations.

6. How is Erlang used at MochiAds? If you can, describe how Erlang
applications are deployed.

It’s easier to list the things that we don’t use Erlang for: data
warehousing (SQL), ETL (batch processing), and the web interface (what
game developers interact with). All of that is PostgreSQL and Python
based.

Erlang powers almost all of our internal tools (e.g. monitoring), the
backend of our MochiBot analytics product, our single-sign on system,
and it powers the ad servers and the supporting technology for them.
Python based servers cater to thousands of developers (at most) per
day, but millions of game players end up talking to our Erlang servers
one way or another every day.

The short of deployment is that we check our ebin files into
subversion, and we issue a command into one of our servers (any of
them) that does a rolling svn update on all of its sibling nodes and
reloads a given list of modules in a specific order. The way we set up
our nodes is that each “application” has a distinct node prefix, e.g.
‘mochiads@’ and a sibling node is defined as any other node that has
the same prefix (’mochiads@10.0.0.1′, ‘mochiads@10.0.0.2′, …). We
deploy some of the applications on every machine in our data center
(supporting infrastructure for distributed computing and monitoring,
mostly), and some of them we deploy on a smaller number (the
performance hungry ones such as the MochiAds and MochiBot servers).

It’s definitely a much less rigorous process than the OTP release
cycle and we don’t do anything fancy like having diskless nodes, but
we’ve found that this is a good balance of robustness, ease of use,
and it doesn’t get in our way when we need to make a quick update. If
I want to change how something works on our servers, all I have to do
is check something in and issue a short and easy to remember command.

7. MochiAds is hiring — how hard has it been to find Erlang-savvy
programmers?

So far there are four of us (including myself) writing Erlang. None of
them had any significant Erlang experience prior to working for Mochi
Media, but they were all smart guys that I had a good deal of
confidence in. I suppose it was a bit risky, but nobody has had any
problems getting into it, and they seem to prefer Erlang to anything
else when starting new projects. Everyone on my team has gone from
negligible Erlang experience to useful code in less than a week
without a whole lot of training or hand-holding.

What I’m finding really interesting is that Erlang is a really big
selling point for us when hiring. There are some VERY smart people out
there that want to do Erlang programming professionally, and some of
them are actually coming directly to us. This is more or less unheard
of around here in San Francisco in this market. Hiring good talent
takes a while, but there are a couple people we’re courting right now
and I think we’re on-track despite Erlang being such a niche language.

8. Any predictions for Erlang in 2008?

It’s hard to predict what other people are going to do, but we’ll
probably get around to properly open sourcing mochiweb (our web
server) and maybe some other in-house components. I’m looking forward
to R12, especially for bit-level binaries and binary comprehensions.
I’d like to see good things happen around CEAN, but I don’t want to
jinx it :)

9. Do you speak any Swedish? ;-)

I used to work with a Swedish designer and he said “aova” a lot.
That’s likely misspelled or slang but I think it means “cool”. I have
another friend that’s really into Kent so I’ve heard some Swedish
music, but not often enough to have memorized the lyrics.

I’ve never been to Sweden, but I was originally planning to go to EUC
this year. Unfortunately it turns out that I’m going to be at a gaming
conference in Seoul at the same time… so maybe next year!