multiprocessing.Pool and KeyboardInterrupt

Someone pinged me recently - he was having a problem handling a keyboard interrupt when using multiprocessing.Pool. Fundamentally, the problem comes with the fact that if a process gets an exception, it just throws the exception instead of returning or becoming .join()-able by the parent. You want to use pool.terminate() in the main function - but you also need to capture and handle the exception in the child so that when then exception is caught - the process can exit "gracefully":

import multiprocessing
import time

def create():
    try:
        time.sleep(10)
    except KeyboardInterrupt:
        return 
    return 'hi mom'

def main():
    def cb(what):
        print what

    pool = multiprocessing.Pool(2)
    try:
        for i in range(2):
            pool.apply_async(create, args=(), callback=cb)
        pool.close()
        pool.join()
    except KeyboardInterrupt:
        print 'control-c presd butan'
        pool.terminate()

if __name__ == "__main__":
    main()

If the exception is thrown within the child, and it just raises - it becomes unmanageable by the parent. As you can see from this piece of code (from pool.py, python trunk) the Pool is attempting to join() the child - which is of course, unjoinable (sort of like me, when I was a teenager):

    @classmethod
    def _terminate_pool(cls, taskqueue, inqueue, outqueue, pool,
                        task_handler, result_handler, cache):
        ...
        if pool and hasattr(pool[0], 'terminate'):
            debug('joining pool workers')
            for p in pool:
                p.join()