Benchmark followup: Google-code Edition

September 16th, 2007 § 4 comments

Since last weeks tem­pest in a blog­pot, and my sub­se­quent post “Have GIL: Want Bench­marks” I’ve been doing a lot of read­ing, plan­ning and dis­cus­sion with people.

What this has cul­mi­nated in is a google-code project I started late last week, tap­ping Daniel Watkins and pick­ing the brains of oth­ers to start pulling together infor­ma­tion about the entire threaded/concurrency domain.

The google-code project is named “python-distributed” and right now, has Daniel Watkins ini­tial work, plus a bunch of wiki pages with my research in it. It is not exhaus­tive by any means — I’ve only just started tak­ing half-baked blog posts, notes scat­tered all over my hard drive and code that I’ve got an try­ing to orga­nize it into something.

I know peo­ple want “num­bers”. I know peo­ple want hard, fast and deci­sive num­bers — that’s what bench­marks are, right? More fre­quently, bench­marks are a light­ning rod of con­tro­versy. Some will yell fowl, oth­ers will trum­pet them as the great­est thing since “Good Night Moon” (truly, a god send). The only good bench­mark is the one with­out an ulte­rior motive, and with the infor­ma­tion out in the open so that every­one can run it.

I like to think of “num­bers” being a non-goal for this project — per­son­ally, I want to explore all of the concurrency-like and true concurrency/threading/etc pack­ages out there for python with a sim­ple set of base­line tests. The even­tual goal is to find one (or build one) that will make it into the python-lib the library must be easy to use, safe, and pythonic.

I’ve spo­ken to many peo­ple and my goal is to gen­er­ate a lot of tests — my pri­mary back­ground is in test engi­neer­ing (automa­tion) — so lay­ing out what code tests we want, what pack­aged we want to test and how we want to test them is pri­mary in my mind. I mean, just the chance to visit all of these pack­ages and ideas and hack up some code, deli­cious, deli­cious code.

So, with that all said: I’m sure peo­ple want to con­tribute, review, etc. With any luck I’ll be able to start push­ing in real code shortly, and Daniel has already pushed in his ini­tial test code to his sand­box.

If you want to add-to, con­tribute, any­thing — feel free to shoot me an email to ask me to add you as a con­trib­u­tor44.4

  1. Again, the project is here444

4

  • Pingback: jessenoller.com - Have GIL: Want Benchmarks.

  • http://blog.dscpl.com.au Gra­ham Dumpleton

    Please also con­sider ref­er­enc­ing the exist­ing solu­tions for Python web frame­works that have been round for many years. In par­tic­u­lar, multi process web servers such as Apache where Python can be embed­ded within the server processes using mod_python or mod_wsgi. Other pos­si­ble solu­tions are dae­mon mode of mod_wsgi, mod_fastcgi, mod_fcgid and mod_scgi mod­ules for Apache. These allow mul­ti­ple dae­mon processes to be cre­ated and to some degree man­aged by Apache, and to which HTTP requests can be prox­ied for pro­cess­ing. Other web servers such as lighttpd and nginx also sup­port fastcgi functionality.

    Because of the way the web servers can dis­trib­ute requests across mul­ti­ple processes, one can very eas­ily make use of mul­ti­ple proces­sors and cores with no change to applications.

    For my orig­i­nal response to com­ments made by Guido in this Par­al­lel Python dis­cus­sion out­lin­ing these solu­tions see:

    http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html

    An older blog entry:

    http://blog.dscpl.com.au/2007/07/web-hosting-landscape-and-modwsgi.html

    also describes why the Python GIL is not as big a deal in Python web appli­ca­tions hosted under Apache as some think it is.

  • http://www.jessenoller.com jesse

    Thanks for the infor­ma­tion — one of the key things that was brought up to me in pri­vate is to make sure that if “we” found “some­thing” that was attrac­tive, that we should make sure that projects like web.py/django/TG/etc could take advan­tage of it.

    As for:

    “Because of the way the web servers can dis­trib­ute requests across mul­ti­ple processes, one can very eas­ily make use of mul­ti­ple proces­sors and cores with no change to applications.“

    You’re dead on — and I had for­got­ten that very valid point in this discussion.

    I’ve added your infor­ma­tion and links to the wiki

  • http://www.dougma.com/ Doug Napoleone

    Quite off topic, but I would rec­om­mend ‘If you give a Moose a Muf­fin’ as well as the other books by that author. (Oh and the entire series from the author of ‘Click Clack Moo’.)

What's this?

You are currently reading Benchmark followup: Google-code Edition at jessenoller.com.

meta