Async Network Programming

I’m digging into Twisted’s async programming model and trying to wrap my head around the core bits of it. I had one of those “aha!” moments tonight where a big chunk of the architecture just came together for me. So far, my experience with Twisted has been like the ESR Lisp quote (which I’ll now mangle): “Learn Lisp even if you’ll never use it. It will make you smarter and a better programmer.”

I’ve gained a new perspective on how to write really efficient networking code using async techniques. I’m thinking that the patterns in Twisted would be useful in other languages, like Java, which lack a completely developed async library. Java throws you to the wolves with the NIO package or you’re stuck with the thread-per-socket model which doesn’t really scale all that well and can perform like a sedated snail on a cold day.

Enough of a tangent. My “aha!” moment tonight was when I finally got how Deferreds are used in Twisted. I don’t know why I didn’t get it at first since the documentation practically mentions it several times. Basically, any time code might need to perform some sort of blocking IO, like reading or writing to a socket, you should use a deferred. The read or write will occur in the run loop of the running reactor. When the read/write completes the code can then trigger a callback. For example:


class Consumer:
def receiveData(self, data):
# do something with the data

def fetchData(self):
fetcher = Fetcher()
deferred =defer.Deferred()
deferred.addCallback(self.receiveData)
deferred.addErrback(self.fetchDataError)
fetcher.readData(deferred)
class Fetcher:
def readData(self, deferred):
try:
data = socket.read()
deferred.callback(data)
except socket.error, se:
deferred.errback(se)


Elegant and useful.