Dead Simple Process Pooling

The pool module provide a really simple, but useful, process pooling and load balancing mechanism. pool:start/1 is used to start worker Erlang nodes and pool:pspawn/3 or pool:pspawn_link/3 is used to queue work items into the pool.

The thing I really like about pool is it handles code loading transparently. Meaning that the worker nodes can access all the code the master node has available on its code path. This makes setting up a process pool even easier since there’s no per-worker configuration needed.

At the bottom of the post is the source for a Erlang module which uses pool to calculate fibnoacci numbers. Highly useful, I know. But sometimes the classic examples are the best :) A quick word of thanks to fellow Erlang Studio attendee Noah Thorp for his efficient fib implementation.

There are four easy steps to using the fibber module:

1. Configure the master node for disributed Erlang and setup the required code paths like so: erl -sname foo -cookie erlang -pa .
2. Call fibber:start/1 with the desired number of workers nodes.
   Example:

   (foo@perdido)1> fibber:start(3).
   Started worker3
   Started worker2
   Started worker1
   ok
   (foo@perdido)2>
   

   If you’re running a Unix-like operating system, you can use the following command to verify that master + # of workers Erlang processes are running: ps -A | grep beam | grep -v "grep beam" | wc -l

3. Start calculating fibonacci numbers by calling fibber:calc:

   (foo@perdido)2> {Node, Result} = fibber:calc(15).
   {worker1@perdido,610}
   (foo@perdido)3> Node.
   worker1@perdido
   (foo@perdido)4> Result.
   610
   

4. When you’re all done using fibber, call fibber:stop() to shutdown the process pool.

No other language I’ve ever used, including Java, Python, Ruby, or C++, has made thread/process pooling so easy or flexible.

-module(fibber).

-export([calc/1, calc/2, server_calc/2, start/1, stop/0]).

start(0) ->
	ok;

start(Workers) ->
	Name = list_to_atom("worker" ++ integer_to_list(Workers)),
	pool:start(Name),
	io:format("Started ~p~n", [Name]),
	start(Workers - 1).

stop() ->
	pool:stop().

calc(N) ->
	fibber:calc(N, 10000).

calc(N, Timeout) ->
	pool:pspawn(fibber, server_calc, [self(), N]),
	receive
		{reply, Result} ->
			Result;
		Wtf ->
			io:format("Didn't expect ~p~n", [Wtf])
	after Timeout ->
		timeout
	end.

server_calc(Caller, N) ->
	Result = fib(N),
	Caller ! {reply, {node(), Result}}.

fib(1) -> 1;
fib(2) -> 1;
fib(N) when N > 2 -> fib1(N,1,1).

fib1(3,P1,P2) -> P1 + P2;
fib1(N,P1,P2) ->
    fib1(N-1,P2, P1 + P2).