jessenoller.com

I saw this on the ‘tubes today: Con­cur­rency with Python, Twisted, and Flex by Bruce Eckel.

Ever since I read it, I’ve got­ten that tickle again to tackle twisted. He offers a pretty ele­gant solu­tion to the prob­lem of chunk­ing up the work and spread­ing it across CPUs.

The draw for me to his solu­tion is the asyn­chro­nous way in which the results are dispatched/captured (I’m going to be doing a lot of async work soon). It’s a pretty good mini-intro into the semi-unpenetrable fort of Twisted.

Per­son­ally, I would have prob­a­bly approached this using the pro­cess­ing mod­ule ala (note, I assumed access to Bruce’s methods):

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from processing import Pool, TimeoutError
pool = Pool(detectCPUs())
 
# ... skipping building the worklist, in bruce's case it's $cores * steps
TASKS = [ step1, step2, step3 ]
 
results = [pool.apply_async(t for t in TASKS]
imap_it = pool.imap(TASKS)
imap_unordered_it = pool.imap_unordered(TASKS)
 
print 'Ordered results using pool.apply_async():'
for r in results:
    print '\t', r.get()
print

Note, of course — I lifted the above right from the doc­u­men­ta­tion exam­ples for the mod­ule. Yes — I know, it lacks the network-ability that Bruce’s twisted-based solu­tion has innately (which does make his solu­tion quite nice). You could do the same with the pro­cess­ing Client/Listeners in the­ory, but twisted is doing all of the con­nec­tion man­age­ment for him under the covers.

Dan­git Bruce gave me the twisted itch again.

And later I’ll do a more com­plete solu­tion with the pro­cess­ing mod­ule, and com­pare the speeds of both against one another. I pre­fer the single-module solu­tion over the twisted.* approach — but I can be swayed. :)

I saw this on the 'tubes today: Concurrency with Python, Twisted, and Flex by Bruce Eckel. Ever since I read it, I've gotten that tickle again to tackle twisted. He offers a pretty elegant solution to the problem of chunking up the work and spreading it across CPUs. The draw for me to his solution ...