Popen in a function is blocking that function from being executed in a separate thread
<p>I'm trying to execute a binary from a python program (under Linux) and would like to dedicate this to a separate thread. Edit: The binary will run for quite a while and write data to disk. It would be nice if I could retreive stdout and stderr in python so that I can write this to a log. It would also be nice if I could evaluate the return value of the binary to make sure that it succeeds. </p>

<p>However, as soon as the function that I start with the thread contains the Popen statement, the thread doesn't seem to start at all :-( Is that a weird thing about thread safety that I don't understand? As soon as I comment out the part where the external command is called, the function is called fine....</p>

<p>Here is a minimum version of my code, I'm using python 2.7.9:</p>

<pre><code>from subprocess import Popen, PIPE
import subprocess
import sys
from threading import Thread


def record ( _command):
    print "Starting function"
    print "Now executing the following command: " + _command
    sys.stdout.flush()

#Here starts the interesting part. As long as this parts is part of the function, the function is not started at all.
#Not even the two lines before are executed :-(
#If I comment out the following lines, at least the print statements work
    process = Popen([_command], stdout=PIPE, stderr=PIPE, shell=True)
    stdout_lines = iter(process.stdout.readline, "")
    for stdout_line in stdout_lines:
        yield stdout_line

    process.stdout.close()
    return_code = process.wait()
    if return_code != 0:
        raise subprocess.CalledProcessError(return_code, _command)


###########################
#Main function starts here#
###########################
threads=[]
for i in range (0,5):
    t=Thread(target=record,args=('ls',))
    threads.append(t)
    t.start()

# wait for all threads to finish
for t in threads:
    t.join()
print "Exiting skript"
</code></pre>

<p>This code just prints out:</p>

<pre><code>Exiting skript
</code></pre>

<p>As soon as I comment out the Popen part, I get:</p>

<pre><code>Starting function
Now executing the following command: ls
Starting function
Now executing the following command: ls
Starting function
Now executing the following command: ls
Starting function
Now executing the following command: ls
</code></pre>

<p>Starting function
    Now executing the following command: ls</p>

<pre><code>Exiting skript
</code></pre>

<p>Your <code>record</code> function doesn't work as intended not because of the <code>Popen</code> call but because it's a generator function (because it contains a <code>yield</code> statement). Generator functions don't actually run their code when you call them. Instead, they return a generator object immediately, and the code in the function is run only when the generator object is iterated on.</p>

<p>I don't think there's any useful way to call a generator function directly as the target of a thread. Instead, you might want the thread to target some other function that consumes a generator object. Or you could restructure your current function to not be a generator (it's not obvious what the purpose of the <code>yield</code> statements was).</p>

python multithreading