Worse Is Better Scaling

Posted: October 11th, 2007 | Author: kevin | Filed under: Programming | View Comments

I’ve been keeping up with the conversations around Tim Bray’s WideFinder project. It’s spawned a couple of interesting threads on erlang-questions and several insightful blog posts

Aloof Schipperke (hopefully not his real name) observations on the possible sad short-term future of scaling caught my eye. Here’s the relevant bit:

At the risk of sounding like a pessimist, I think we’ll end up with thousands of little SOA web services engines. Each one handling a single piece. Each one with its own HTTP stack. Each one using PHP/Perl/Ruby/etc to implement the service functions. Each one sitting on top of a tiny little mysql database.

This is how I see development shops today approaching problems of scale. That’s not to say that modular, loosely coupled subsystems aren’t a good thing because they are. The part I find scary is the approach to spinning up multiple copies of stacks (HTTP, application, database, etc.) in order to run these subsystems. It seems resource wasteful and an absolute bear to manage.

I know that there is a fair amount of resistance to Erlang for a lot of different reasons. Whether or not Erlang will be a mainstream language is unclear to me. Even if Erlang doesn’t go mainstream I really hope that some of the core Erlang memes like “writing distributed code should be easy” or “concurrency is cheap and easy” propagate into whatever becomes popular. I have a hard time seeing how these memes can be adopted by the current crop of mainstream languages but I’m not an expert and there are a lot of smart people work on this problem.

I’m not going to wait for the mainstream, though. Like I’ve said before, Erlang suits my needs just fine. I’ll be using it while I watch the mainstream figure out how to adopt these concepts.

  • I'm definitely of two minds when it comes to the topic of LAMP. On one hand, the management and provisioning of large sets of LAMP stacks is still problematic. On the other hand, it seems an inevitable.

    As Anders mentions, the overhead can be relatively small, particularly if some care is exercised in the design. Having said this, there are definitely problem sets where resorting to loosely coupled LAMP stacks is inappropriate.

    Regarding manageability, I agree completely. In fact, this is the crux of my concern. Horizontal scaling of LAMP stacks will probably require substantial simplification and automation. Otherwise we'll see another wave of vendor products that purport to solve the problem while only serving to add more complexity. Hopefully we'll see improvements in these areas before the inevitable happens...

    Erlang in its current form is probably not suited as a mainstream tool, but it's nice to see it gaining popularity. I sometimes wonder if a future derivation could break out of the 'exotic language' category. The question would then become, what problems created by massively parallel LAMP stacks will be well suited for concurrent languages?
  • I've been embracing this approach for a while now of "little SOA web services engines" and have been referring to it as "microapps". It's how I build pretty much everything these days. I haven't found it to be terribly wasteful in terms of resources; my clusters of microapps (mostly written in Python and Perl) still seem to be much lighter weight than an equivalent java stack would be. Management is an issue though and has required some extra automation work in that area and could still stand to have a lot more. I don't think it's insurmountable though.

    I like Erlang a lot and studying Erlang/OTP has heavily influenced how I put microapps together. I hope to write as much of my future code as possible in Erlang. However, I still see a lot of advantages to keeping things as loosely coupled separate applications communicating over HTTP. Ultimately, I have no loyalty. I'm currently most productive in Python at the moment and think it's a great language, but I'd rather not commit to it any more than I have to. Erlang might succeed in romancing me away from Python, but I won't want to commit to that any more than I have to either. Languages come and go and have strengths and weaknesses in different domains. As pretty and shiny as any one might be, I'd rather not be stuck with it forever. The bigger the application you build, the more expensive it gets to switch to another language or platform later on. If I decide tomorrow that I don't want to write another line of Python and go all Erlang, I can still use all or most of the stuff I've already written in Python (and Perl before that). When I move on to another language after Erlang, I can keep using all of that code, as long as the new language has HTTP libraries. If some other language comes out with a really powerful solution to a problem I need to deal with, I can write one microapp in that language and have it accessible from the others without having to switch over wholesale just to take advantage of one feature or really nice library. I have very little faith in any one language or platform, but I'm fairly confident that HTTP will be a big part of the programming landscape for a long time to come.
blog comments powered by Disqus