Skip to content

multiprocessing.Pool and KeyboardInterrupt

Someone pinged me recently – he was having a problem handling a keyboard interrupt when using multiprocessing.Pool.

Fundamentally, the problem comes with the fact that if a process gets an exception, it just throws the exception instead of returning or becoming .join()-able by the parent. You want to use pool.terminate() in the main function – but you also need to capture and handle the exception in the child so that when then exception is caught – the process can exit “gracefully”:

?View Code PYTHON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import multiprocessing
import time
 
def create():
    try:
        time.sleep(10)
    except KeyboardInterrupt:
        return 
    return 'hi mom'
 
def main():
    def cb(what):
        print what
 
    pool = multiprocessing.Pool(2)
    try:
        for i in range(2):
            pool.apply_async(create, args=(), callback=cb)
        pool.close()
        pool.join()
    except KeyboardInterrupt:
        print 'control-c presd butan'
        pool.terminate()
 
if __name__ == "__main__":
    main()

If the exception is thrown within the child, and it just raises – it becomes unmanageable by the parent. As you can see from this piece of code (from pool.py, python trunk) the Pool is attempting to join() the child – which is of course, unjoinable (sort of like me, when I was a teenager):

?View Code PYTHON
1
2
3
4
5
6
7
8

    @classmethod
    def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
                        task_handler, result_handler, cache):
        ...
        if pool and hasattr(pool[0], 'terminate'):
            debug('joining pool workers')
            for p in pool:
                p.join()
  • Casey
    What and how is _terminate_pool supposed to be used? Can you add some more detail?
  • Joe
    If you change range(2) to range(200) it does not exit correctly. How would you solve this?
  • Shimmer
    I tried your example and got errors without the nice messages as expected:

    None
    None
    Traceback (most recent call last):
    File "testpool.py", line 26, in <module>
    main()
    File "testpool.py", line 23, in main
    pool.join()
    File "C:\Python26\lib\multiprocessing\pool.py", line 340, in join
    self._result_handler.join()
    File "C:\Python26\lib\threading.py", line 634, in join
    self.__block.wait()
    File "C:\Python26\lib\threading.py", line 237, in wait
    waiter.acquire()
    KeyboardInterrupt
  • Sorry, I altered the code after running it, I updated the example
  • Aki
    Interesting.
    I did it differently.
    I disabled SIGINT to any child process using signal.signal(signal.SIGINT, signal.SIG_IGN)
    The parent takes SIGINT then send a message (or set an event) to each child asking it to shoudown.
  • Jean-Paul
    Why doesn't apply_async have a means for receiving notification of errors from the child process? This isn't just related to C-c, right? Any exception raised in the child will cause the process to go into an unmanageable state - that seems like a terrible way to deal with errors.
  • See above: exceptions within the children are handled properly in some cases.
  • I don't disagree. I'm thinking about how to change it under the covers. I may file an enhancement against myself to track it.
  • Jean-Paul
    Great. :)
  • Actually, let me amend this. The library *does* correctly handle exceptions thrown in the child - if you modify the example in my post and just add a broken import (don't hit control-c) the pool will exit gracefully.
  • gaius.julius
    anyways, what should I do to gracefully handle KeyboardInterrupt and other similar cases when using Pool?
  • See the main post. A keyboard interrupt is not handled the same as other exceptions. At least in py2.5-forward the solution in my main blog post works.
  • Ben
    You almost made me laugh with that joke... almost...
  • Nerd humor for the loose.
blog comments powered by Disqus