Thoughts on EC2 & virtuerl

Posted: January 28th, 2008 | Author: kevin | Filed under: Erlang | View Comments

I’ve been spending the last few evenings working on an Erlang AMI image. I want to make deploying Erlang nodes into EC2 as easy as possible. A quick perusal of the public images didn’t yield what I was looking for so I’ve been digging in and learning the ins and outs of EC2 and S3. There are a few rough edges but I think Amazon has done a great job overall.

When I’m done the image will make it drop-dead simple to deploy Erlang nodes into EC2’s grid. I’m basing the image on Ubuntu 7.10 server and the latest Erlang sources along with a few changes to allow distributed Erlang to work over a single SSL socket (hat tip to Michael McDaniel for the excellent wiki article). The image will be free and publicly available once I’ve shook out most of the bugs.

The next thing I want to do is create infrastructure to harness the power of Erlang nodes on EC2. That’s where virtuerl and MochiWeb come into the picture. MochiWeb is a web server construction kit of sorts. I intend to use it to create a “Erlang aware” proxy which delegates requests to the nodes running inside EC2 like so:

The proxy will use virtuerl to automatically start and stop nodes as needed, within limits, to handle the requests.

virtuerl, when it’s all grown up, will be much more than a wrapper around the EC2 APIs. virtuerl will be a full-fledged management library for EC2 nodes. My goal is to be able to start a node and remotely deploy an application onto it in just a few lines of code. Of course virtuerl will cooperate with existing Erlang features like the code server to make deployment as compatible as possible with existing Erlang applications. Why reinvent tools when the language comes with such nice tools already built in?

I think this design is a compelling way to use the EC2 infrastructure and would be compelling to businesses ranging in size from startups to more established companies. I’m pretty sure Joel Reymont has done something similar before but is prevented from giving the details due to NDAs. So while I’m not the first guy to think of this nor the first one to even do it, my goal is to implement this all out in the open so more projects can leverage the power of grid computing and Erlang.

  • Interesting. I haven't seen that yet, but then again I haven't really put the image thru its paces yet. Thanks for the tip -- this is something I'll definitely look for.
  • Kevin, do you see also in your Ubuntu images something like a "Erlang High Resolution Timer Error" if you start Erlang immediately after EC2 instance start ? AFAIK it is DNS issue and happens on all Debian derived systems and and I just retry (with some sleep between) till it works, it takes usually about 20 seconds, at least on my systems.

    P.S. check your google devpotato mail, I sent you also a private message !
  • Sometimes you just gotta go with what works. If I wanted to take the time I'd figure out how to get a Fedora-based image working but other people have already done the hard work of scripting image building. I just need to modify it for my own uses.

    Standing on the shoulders of those that came before me and all that...
  • Ubuntu? Traitor!
blog comments powered by Disqus