jessenoller.com

Next up in the GBLOSTR (great big list of stuff to review) is the Cir­cuits library by James Mills (here and here) I’m famil­iar only with James Mills’ posts on python-list, but more recently, I know he’s been work­ing on get­ting some level of mul­ti­pro­cess­ing into cir­cuits — cir­cuits was already on my research list, but I bumped it up on the queue because we started chatting.

Let me state this: these are slightly cleaned up ver­sions of my notes as I am learn­ing these mod­ules — some of them have higher learn­ing curves for me than others.

Cir­cuits is, well — an event based “frame­work” (again, note the small f) based around the con­cept of Com­po­nents (big C!) consuming/reacting and in turn gen­er­at­ing events — all asynchronously.

James’ goals seem pretty sim­ple — build some­thing with no exter­nal depen­den­cies, that’s com­pact (I should say, the core.py is < 500 lines) and that makes it easy to build scal­able mes­sag­ing based (event based) sys­tems. Did he suc­ceed — don’t know! Let’s dig in.

» Read the rest of this entry «

Next up in the GBLOSTR (great big list of stuff to review) is the Circuits library by James Mills (here and here) I'm familiar only with James Mills' posts on python-list, but more recently, I know he's been working on getting some level of multiprocessing into circuits - circuits was already on my ...

Note:This is the first post in what I hope will be a series lead­ing up to my concurrency/distributed sys­tems talk at PyCon. I’m steadily work­ing through exper­i­ment­ing with and learn­ing the var­i­ous frameworks/libraries in the python ecosystem.

I reserve the right (and prob­a­bly will) to revise these entries based on feed­back from peo­ple (mainly the author(s) of said tool(s)). I will also add addi­tional bits and pieces as I learn and explore more. Code and exam­ples will be checked into my pycon 2009 bit­bucket site here/Note

For awhile now, I’ve been mean­ing to dig into Kamaelia but was largely put off by what I tend to call the “twisted effect”. What this means is that when I go look­ing for libraries and small com­po­nents, I go look­ing for a library — not a “solu­tion”. I also worry about the “once you go in, you must fol­low this par­a­digm” effect. I’m not going to say that these feel­ing con­tinue to be founded, or are com­pletely ratio­nal — after all, I am dig­ging into it, yes? It’s the thought that once you adopt the “one true way of doing things” you’re trapped in that solution/framework “for­ever” — iron­i­cally, I love Django for it’s “con­cep­tual integrity” and full-stack approach. No, I don’t under­stand me either.

» Read the rest of this entry «

Note:This is the first post in what I hope will be a series leading up to my concurrency/distributed systems talk at PyCon. I'm steadily working through experimenting with and learning the various frameworks/libraries in the python ecosystem. I reserve the right (and probably will) to revise these entries based on feedback from people (mainly the ...

Steve Holden shot an email out to python dev last night. In it, he said some­thing which at first tweaked me, but in think­ing about it, I both see, and some part of me agrees with his point. To quote:

I think that both 3.0 and 2.6 were rushed releases. 2.6 showed it in the
inclu­sion (later rec­og­niz­able as some­what ill-advised so late in the
day) of mul­ti­pro­cess­ing; 3.0 shows it in the very fact that this
dis­cus­sion has become nec­es­sary. So we face an impor­tant turn­ing point:
is 3.1 going to be seri­ous pro­duc­tion qual­ity or not?

» Read the rest of this entry «

Steve Holden shot an email out to python dev last night. In it, he said something which at first tweaked me, but in thinking about it, I both see, and some part of me agrees with his point. To quote: I think that both 3.0 and 2.6 were rushed releases. 2.6 showed it in the inclusion ...

Via David Goodger / py-dev

Reg­is­ter here:
http://us.pycon.org/2009/register/

Infor­ma­tion (rates etc.):
http://us.pycon.org/2009/registration/

Hotel infor­ma­tion & reser­va­tions:
http://us.pycon.org/2009/about/hotel/

Don’t know if I am pay­ing my own way, or going on com­pany dime yet — but given I have two talks, I should be going. I’ve con­sid­ered sell­ing my body to sci­ence, but I keep get­ting lowballed.

Via David Goodger / py-dev Register here: http://us.pycon.org/2009/register/ Information (rates etc.): http://us.pycon.org/2009/registration/ Hotel information & reservations: http://us.pycon.org/2009/about/hotel/ Don't know if I am paying my own way, or going on company dime yet - but given I have two talks, I should be going. I've considered selling my body to science, but I keep getting lowballed.

Last night Trent Nel­son dropped a friendly note to the python-committers mail­ing list, you can see the full text here.

He and oth­ers (includ­ing dona­tions from var­i­ous com­pa­nies) have put together a net­work of machines with var­i­ous oper­at­ing sys­tems aimed at giv­ing var­i­ous projects wider (and com­plete, hooray logins!) access to plat­forms we nor­mally don’t have access to in addi­tion to the ones we have, but don’t have login access to.

To quote Trent:

And that’s when it hit me. Build­bots are fine when every­thing is
run­ning smoothly, but noth­ing com­pares to actu­ally hav­ing access to
a sys­tem when you’re try­ing to debug something.

So, I thought to myself, why not buy a cou­ple of clunky old boxes
off eBay and donate them to the PSF, such that all devel­op­ers had
access to them? I dropped a note to Guido and Neal, they put me
in touch with Titus, who had just accepted a posi­tion at Michi­gan
State, and, well…

Ten months, seven trips to MSU, six blown fuses and about $60,000
later, I’m proud to intro­duce you all to Snakebite: The Open Net­work!
A net­work of around 37-ish servers of all dif­fer­ent shapes and sizes,
spread over three sites, specif­i­cally geared towards the needs of
open source projects like Python.

Every CPython, Jython, Iron­Python and PyPy com­mit­ter will have access
to every devel­op­ment server on the net­work. I’ve also extended the
offer to promi­nent Python projects like Django and Twisted.

Even­tu­ally, I’ll invite other open source projects to par­tic­i­pate
(Apache, Sub­ver­sion, MySQL, Post­gres, etc), but the net­work is my
gift to All Things Python, first and fore­most, so Python projects
will always get pref­er­en­tial treatment.

Trent and other have man­aged to begin work on scratch­ing an itch I know a lot of us have felt. Just for mul­ti­pro­cess­ing alone, I have to main­tain a small farm of VMware images (today I’m adding fbsd and open­so­laris) to do bug fixes and devel­op­ment on.

Hav­ing a much larger, ded­i­cated build­bot farm with login priv­i­leges is going to be awe­some. Given that they’re expand­ing this to other projects as well is just icing on a deli­cious, deli­cious cake (mmm, cake).

I had con­sid­ered doing some­thing like this myself; but given I don’t have access to cheap host­ing, or hard­ware — that’s sort of makes it impos­si­ble. I had sketched up a plan to get an Apple 1U machine and throw virtualbox/vmware on it and host as many OSes and accounts as pos­si­ble (OS/X is cur­rently the only OS with­out rep­re­sen­ta­tion in the machines cur­rently listed).

Really, this is fan­tas­tic and the work and time that’s gone into putting this together as well as the dona­tions com­pa­nies like HP, Microsoft and oth­ers have made is sim­ple great.

As Trent notes in his email — they’re run out of phys­i­cal room to host stuff right now, but I have to imag­ine that as word spreads more com­pa­nies and peo­ple will gladly donate time and resource to what is really an excel­lent project. VPS hosts could prob­a­bly also get in the game in time as well.

This is just awesome.

You can see the site they’ve put together at http://www.snakebite.org/ and http://www.snakebite.org/network

Addi­tion­ally, Steve and Titus have posts too.

Last night Trent Nelson dropped a friendly note to the python-committers mailing list, you can see the full text here. He and others (including donations from various companies) have put together a network of machines with various operating systems aimed at giving various projects wider (and complete, hooray logins!) access to platforms we normally ...

As noted on the PyCon blog, the offi­cial list of talks has been posted. There’s a hell of a lot I want to go to. I might even con­sider going to the two talks I’m presenting.

The list of talks is here.

As noted on the PyCon blog, the official list of talks has been posted. There's a hell of a lot I want to go to. I might even consider going to the two talks I'm presenting. The list of talks is here.

Suf­fer­ing from insom­nia this morn­ing, I decided to delve into my python-list archive box in gmail. I nor­mally only scan it once or twice a month due to sig­nal to noise ratio. A post by James Mills here caught my eye:

I’ve noticed over the past few weeks lots of ques­tions
asked about multi-processing (includ­ing myself).

For those of you new to multi-processing, per­haps this
thread may help you. Some things I want to start off
with to point out are:

“mul­ti­pro­cess­ing will not always help you get things done faster.”

“be aware of I/O bound appli­ca­tions vs. CPU bound”

“mul­ti­ple CPUs (cores) can com­pute mul­ti­ple con­cur­rent expres­sions -
not read 2 files concurrently”

“in some cases, you may be after dis­trib­uted pro­cess­ing rather than
multi or par­al­lel processing”

cheers
James

» Read the rest of this entry «

Suffering from insomnia this morning, I decided to delve into my python-list archive box in gmail. I normally only scan it once or twice a month due to signal to noise ratio. A post by James Mills here caught my eye: I've noticed over the past few weeks lots of questions asked about multi-processing (including myself). For ...

Some­one pinged me recently — he was hav­ing a prob­lem han­dling a key­board inter­rupt when using multiprocessing.Pool.

Fun­da­men­tally, the prob­lem comes with the fact that if a process gets an excep­tion, it just throws the excep­tion instead of return­ing or becom­ing .join()-able by the par­ent. You want to use pool.terminate() in the main func­tion — but you also need to cap­ture and han­dle the excep­tion in the child so that when then excep­tion is caught — the process can exit “gracefully”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import multiprocessing
import time
 
def create():
    try:
        time.sleep(10)
    except KeyboardInterrupt:
        return 
    return 'hi mom'
 
def main():
    def cb(what):
        print what
 
    pool = multiprocessing.Pool(2)
    try:
        for i in range(2):
            pool.apply_async(create, args=(), callback=cb)
        pool.close()
        pool.join()
    except KeyboardInterrupt:
        print 'control-c presd butan'
        pool.terminate()
 
if __name__ == "__main__":
    main()

If the excep­tion is thrown within the child, and it just raises — it becomes unman­age­able by the par­ent. As you can see from this piece of code (from pool.py, python trunk) the Pool is attempt­ing to join() the child — which is of course, unjoin­able (sort of like me, when I was a teenager):

1
2
3
4
5
6
7
8

    @classmethod
    def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
                        task_handler, result_handler, cache):
        ...
        if pool and hasattr(pool[0], 'terminate'):
            debug('joining pool workers')
            for p in pool:
                p.join()

Someone pinged me recently - he was having a problem handling a keyboard interrupt when using multiprocessing.Pool. Fundamentally, the problem comes with the fact that if a process gets an exception, it just throws the exception instead of returning or becoming .join()-able by the parent. You want to use pool.terminate() in the main function ...

I just checked in and pushed a new ver­sion to pypi — in this case, I was really jonesing for the abil­ity to push tests I wrote that imported the module/config dict through pychecker.

This wasn’t pos­si­ble due to the load­ing system’s reliance on nose being the caller — so I added some envi­ron­ment check­ing to it to autoload. See the docs on pypi

I just checked in and pushed a new version to pypi - in this case, I was really jonesing for the ability to push tests I wrote that imported the module/config dict through pychecker. This wasn't possible due to the loading system's reliance on nose being the caller - so I added some environment ...