Popen in a function is blocking that function from being executed in a separate thread
<p>I'm trying to execute a binary from a python program (under Linux) and would like to dedicate this to a separate thread. Edit: The binary will run for quite a while and write data to disk. It would be nice if I could retreive stdout and stderr in python so that I can write this to a log. It would also be nice if I could evaluate the return value of the binary to make sure that it succeeds. </p>

<p>However, as soon as the function that I start with the thread contains the Popen statement, the thread doesn't seem to start at all :-( Is that a weird thing about thread safety that I don't understand? As soon as I comment out the part where the external command is called, the function is called fine....</p>

<p>Here is a minimum version of my code, I'm using python 2.7.9:</p>

<pre><code>from subprocess import Popen, PIPE
import subprocess
import sys
from threading import Thread


def record ( _command):
    print "Starting function"
    print "Now executing the following command: " + _command
    sys.stdout.flush()

#Here starts the interesting part. As long as this parts is part of the function, the function is not started at all.
#Not even the two lines before are executed :-(
#If I comment out the following lines, at least the print statements work
    process = Popen([_command], stdout=PIPE, stderr=PIPE, shell=True)
    stdout_lines = iter(process.stdout.readline, "")
    for stdout_line in stdout_lines:
        yield stdout_line

    process.stdout.close()
    return_code = process.wait()
    if return_code != 0:
        raise subprocess.CalledProcessError(return_code, _command)


###########################
#Main function starts here#
###########################
threads=[]
for i in range (0,5):
    t=Thread(target=record,args=('ls',))
    threads.append(t)
    t.start()

# wait for all threads to finish
for t in threads:
    t.join()
print "Exiting skript"
</code></pre>

<p>This code just prints out:</p>

<pre><code>Exiting skript
</code></pre>

<p>As soon as I comment out the Popen part, I get:</p>

<pre><code>Starting function
Now executing the following command: ls
Starting function
Now executing the following command: ls
Starting function
Now executing the following command: ls
Starting function
Now executing the following command: ls
</code></pre>

<p>Starting function
    Now executing the following command: ls</p>

<pre><code>Exiting skript
</code></pre>

<p>With some small fixes, your code runs:</p>

<ul>
<li>got rid of the generator function, doesn't make sense with threads</li>
<li>simplified the way to get output from the process, but still read line by line (reading the final output with <code>communicate</code> would work but you seem to require having the lines printed immediately to have an idea of the current progress)</li>
<li>protected output using a <code>Lock</code> or it would be mixed up between threads</li>
<li>merged stdout and stderr in pipe to avoid deadlocks</li>
</ul>

<p>fixed code:</p>

<pre><code>from subprocess import Popen, PIPE, STDOUT
import subprocess
import sys
from threading import Thread,Lock

lck = Lock()

def record ( _command):
    lck.acquire()
    print("Starting function")
    print("Now executing the following command: " + _command)
    lck.release()

    process = Popen([_command], stdout=PIPE, stderr=STDOUT, shell=True)
    while True:
        line = process.stdout.readline()
        if len(line)==0:
            break
        lck.acquire()
        sys.stdout.write(line)
        lck.release()

    process.wait()
    return_code = process.wait()
    if return_code != 0:
        raise subprocess.CalledProcessError(return_code, _command)

# main program

threads=[]
for i in range (0,5):
      t=Thread(target=record,args=('ls',))
      threads.append(t)
      t.start()

# wait for all threads to finish
for t in threads:
      t.join()
print("Exiting skript")
</code></pre>

python multithreading