Python multiprocessing silent failure with class
<p>the following does not work using python 2.7.9, but also does not throw any error or exception. is there a bug, or can multiprocessing not be used in a class?</p>

<pre><code>from multiprocessing import Pool

def testNonClass(arg):
    print "running %s" % arg
    return arg

def nonClassCallback(result):
    print "Got result %s" % result

class Foo:
    def __init__(self):
        po = Pool()
        for i in xrange(1, 3):
            po.apply_async(self.det, (i,), callback=self.cb)
        po.close()
        po.join()
        print "done with class"
    po = Pool()
    for i in xrange(1, 3):
        po.apply_async(testNonClass, (i,), callback=nonClassCallback)
    po.close()
    po.join()

    def cb(self, r):
        print "callback with %s" % r

    def det(self, M):
        print "method"
        return M+2


if __name__ == "__main__":
    Foo()
</code></pre>

<p>running prints this: </p>

<pre><code>done with class
running 1
running 2
Got result 1
Got result 2
</code></pre>

<p>EDIT: THis seems related, but it uses <code>.map</code>, while I specifically am needing to use <code>apply_async</code> which seems to matter in terms of how multiprocessing works with class instances (e.g. I dont have a picklnig error, like many other questions related to this) - <a href="http://stackoverflow.com/questions/29009790/python-how-to-do-multiprocessing-inside-of-a-class">Python how to do multiprocessing inside of a class?</a></p>

<p>I'm pretty sure it can be used in a class, but you need to protect the call to <code>Foo</code> inside of a clause like:</p>

<p><code>if name == "__main__":</code></p>

<p>so that it only gets called in the main thread. You may also have to alter the <code>__init__</code> function of the class so that it accepts a pool as an argument instead of creating a pool.</p>

<p>I just tried this</p>

<pre><code>from multiprocessing import Pool

#global counter
#counter = 0

class Foo:
    def __init__(self, po):
        for i in xrange(1, 300):
            po.apply_async(self.det, (i,), callback=self.cb)
        po.close()
        po.join()
        print( "foo" )
        #print counter

    def cb(self, r):
        #global counter
        #print counter, r
        counter += 1


    def det(self, M):
        return M+2


if __name__ == "__main__":
    po = Pool()
    Foo(po)
</code></pre>

<p>and I think I know what the problem is now. Python isn't multi-threaded; global interpreter lock prevents that. Python is using multiple processes, instead, so the sub-processes in the Pool don't have access to the standard output of the main process. </p>

<p>The subprocesses also are unable to modify the variable <code>counter</code> because it exists in a different process (I tried running with the <code>counter</code> lines commented out and uncommented). Now, I do recall seeing cases where global state variables get altered by processes in the pool, so I don't know all of the minutiae. I do know that it is, in general, a bad idea to have global state variables like that, if for no other reason than they can lead to race conditions and/or wasted time with locks and waiting for access to the global variable.</p>

python class multiprocessing