Comments on: Python 3.0, some multiprocessing info, administrative notes.

By: nutrition foods

nutrition foods — Fri, 30 Jan 2009 01:53:43 +0000

Thanks for the info. May God have mercy on us all.

By: Harald Armin Massa

Harald Armin Massa — Tue, 23 Dec 2008 11:54:35 +0000

Jesse,

do you know of anyone who has ported

http://www.cherokee-project.com/download/pyscgi/

pyscgi to use processing / multiprocessing?

SCGI on Windows currently fails to use processes as of missing pickle support for sockets, and multiprocessing / processing explicitly contains them ...

In other words:
to have a "MultiprocessingMixin" within SocketServer.py in standard library would be a GREAT idea :)

Best wishes

Harald

By: Mike Hansen

Mike Hansen — Sat, 06 Dec 2008 01:23:34 +0000

Hi Gael,

Sage has been using multiprocessing from within in IPython for a number of months now. The key thing is that we use some custom code for pickling functions. Usually it's used as a decorator, but you can use it like the following:

sage: f = lambda x: x*x
sage: pf = parallel(2)(f) #two processes
sage: list(pf(range(10)))
[(((0,), {}), 0),
(((1,), {}), 1),
(((2,), {}), 4),
(((3,), {}), 9),
(((4,), {}), 16),
(((5,), {}), 25),
(((6,), {}), 36),
(((7,), {}), 49),
(((8,), {}), 64),
(((9,), {}), 81)]

I'm assuming that you'd want it so that you don't have to have this custom pickling code in there?

--Mike

By: Gael Varoquaux

Gael Varoquaux — Thu, 04 Dec 2008 20:52:47 +0000

Talking about making multiprocessing better, please, please, pretty please with sugar on top, make it work in ipython. Scientific users are very much interested in multiprocessing, but they really rely on ipython for interactive work.
Another nice thing would be support for lambdas (I think IPython1 has achieved this). The number one usecase is to use map on functions with several arguments.

By: Robert Kern

Robert Kern — Thu, 04 Dec 2008 20:40:52 +0000

Something that would help a lot with getting feedback about multiprocessing is a mailing list. Even just a note on the project page saying “Email feedback to Jesse Noller at <…>” would work, but a public, archived mailing list is ideal.

By: jnoller

jnoller — Thu, 04 Dec 2008 20:03:44 +0000

I don't use iPython: So I don't know “what it needs for ipython” – someone
who uses ipython should do that work. As for lambda support: I don't get
what you're saying. How does multiprocessing preclude use of lambdas?

By: jnoller

jnoller — Thu, 04 Dec 2008 19:55:35 +0000

Good idea, generally speaking though, it's in python core – the python
mailing list(s) (python dev, python-list) and the bugtracker are the places
to go. We might be able to add one to the backport list, but that muddies
the water.