How to read stream of a subprocess in python?
<p>I create a process with <code>subprocess.Popen</code> and get its <code>stdout</code>.
I want to read content from <code>stdout</code> and print it out in another thread.
Since my purpose it to make a an interactive program later, I cannot use <code>subprocess.communicate</code>.</p>

<p>My basic requirement is: Once subprocess output something, the thread should immediately print it out to the screen.</p>

<p>Here is the code</p>

<pre><code>import subprocess
import thread
import time

def helper(s):
    print "start",time.gmtime()
    while True:
        print "here"
        print s.next()
    print "finished",time.gmtime()
    thread.start_new_thread(helper,(out_stream,))

def process_main():
    global stdin_main
    global stdout_main
    p = subprocess.Popen("/bin/bash", shell=False, stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT, bufsize=0)
    stdin_main = p.stdin
    stdout_main = p.stdout
    th1 = thread_print_stream(stdout_main)
    stdin_main.write("ls -la\n")
    stdin_main.flush()
    time.sleep(30)

    p.terminate()

process_main()
</code></pre>

<p>Time elapse between "start" and "finished" should be very fast. However it is 30 seconds which is exactly the same as time before process terminated.
I cannot understand why the output is not instance.
Or how can I make it instancely?</p>

<p>As cdarke mentioned. This program is suffered from buffering.
However it is more like a bug in python 2.x.
The logic is nothing wrong in the current program. </p>

<p>To fix the issue, you have to reopen the stdout.
Like the code below:</p>

<pre><code>def thread_print_stream(out_stream):
  def helper(s):
    print "start",time.gmtime()
    for line in io.open(out_stream.fileno()):
      print line
    print "finished",time.gmtime()
  thread.start_new_thread(helper,(out_stream,))
</code></pre>

<p>And add</p>

<pre><code>stdin_main.close()
stdout_main.close()
</code></pre>

<p>before subprocess terminated, to make sure no error rise.</p>

python multithreading subprocess