multiprocessing.Pool and KeyboardInterrupt

by jesse in ,

Someone pinged me recently - he was having a problem handling a keyboard interrupt when using multiprocessing.Pool. Fundamentally, the problem comes with the fact that if a process gets an exception, it just throws the exception instead of returning or becoming .join()-able by the parent. You want to use pool.terminate() in the main function - but you also need to capture and handle the exception in the child so that when then exception is caught - the process can exit "gracefully":

import multiprocessing
import time

def create():
    except KeyboardInterrupt:
    return 'hi mom'

def main():
    def cb(what):
        print what

    pool = multiprocessing.Pool(2)
        for i in range(2):
            pool.apply_async(create, args=(), callback=cb)
    except KeyboardInterrupt:
        print 'control-c presd butan'

if __name__ == "__main__":

If the exception is thrown within the child, and it just raises - it becomes unmanageable by the parent. As you can see from this piece of code (from, python trunk) the Pool is attempting to join() the child - which is of course, unjoinable (sort of like me, when I was a teenager):

    def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
                        task_handler, result_handler, cache):
        if pool and hasattr(pool[0], 'terminate'):
            debug('joining pool workers')
            for p in pool: