Multiprocessing in Python crashes when code reaches start()
<p>I am new in Python. I tried to use some multiprocessing to make my work faster. First I tried an example and everything worked fine. Here is the code:</p>

<pre><code>from multiprocessing import Process
import time

def f(name, n, m):
    if name == 'bob':
        time.sleep(2)
    print 'hello', name, ' ', n, m

def h():
    g(1, 2, 3)

def g(a, s, d):
    p = Process(target=f, args=('bob', a, s,))
    t = Process(target=f, args=('helen', s, d,))
    p.start()
    t.start()
    t.join()
    p.join()
    print("END")

if __name__ == '__main__':
    print("Start")
    h()
</code></pre>

<p>After that I used the same technique to my code and errors appeared. This is a part of the problematic code:</p>

<pre><code>if __name__ == "__main__":
    night_crawler_steam()

def night_crawler_steam():
    .
    .
    .
    multi_processing(max_pages, url, dirname)
    .
    .
    .

def multi_processing(max_pages, url, dirname):
    page = 1
    while page &lt;= max_pages:
        my_url = str(url) + str(page)
        soup = my_soup(my_url)
        fgt = Process(target=find_game_titles, args=(soup, page, dirname,))
        fl = Process(target=find_links, args=(soup, page, dirname,))
        fgt.start() #&lt;-----------Here is the problem
        fl.start()
        fgt.join()
        fl.join()
        page += 1

def find_links(soup, page, dirname):
.
.
.

def find_game_titles(soup, page, dirname):
.
.
.
</code></pre>

<p>When the interpreter reaches fgt.start() some errors appear:</p>

<pre><code>Traceback (most recent call last):
  File "C:/Users/ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½/Desktop/MY PyWORK/NightCrawler/NightCrawler.py", line 120, in &lt;module&gt;
    night_crawler_steam()
  File "C:/Users/ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½/Desktop/MY PyWORK/NightCrawler/NightCrawler.py", line 23, in night_crawler_steam
Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
    multi_processing(max_pages, url, dirname)
  File "C:/Users/ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½/Desktop/MY PyWORK/NightCrawler/NightCrawler.py", line 47, in multi_processing
    fgt.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1384, in load
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 425, in save_reduce
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 864, in load
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 886, in load_eof
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    raise EOFError
    save(v)
EOFError
</code></pre>

<p>This goes on and on till <code>RuntimeError: maximum recursion depth exceeded</code></p>

<p>Any idea would be helpful!</p>

<p>There seems to be a problem with <a href="https://docs.python.org/2/library/pickle.html" rel="nofollow">pickling</a> <code>soup</code> (see <a href="https://docs.python.org/2/library/multiprocessing.html#programming-guidelines" rel="nofollow">Programming Guidelines</a>) so a simple solution would be to move the call <code>my_soup(my_url)</code> into the target functions like so:</p>

<pre><code>def multi_processing(max_pages, url, dirname):        
    p=Pool() # using a pool is not necessary to fix your problem
     for page in xrange(1,max_pages+1):
        my_url = str(url) + str(page)
        p.apply_async(find_game_titles, (my_url, page, dirname))
        p.apply_async(find_links, (my_url, page, dirname))
    p.close()
    p.join()

def find_links(url,page, dirname):
    soup=my_soup(url)
    #function body from before


def find_game_titles(url, page, dirname):
    soup=my_soup(url)
    #function body from before
</code></pre>

<p>(Of course you can also pass soup in a pickle-able format but depending on what <code>my_soup</code> does exactly it might or might not be worth it.)</p>

<p>While not totally necessary it is normal to put the <code>if __name__=="__main__":</code> part in the end of the file.</p>

<p>Also you migth want to have a look at <a href="https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.pool" rel="nofollow">multiprocessing.Pool</a>'s other methods as they may fit better, depending on what your functions do. </p>

python python-2.7 multiprocessing python-multiprocessing