Python subprocess.Popen poll seems to hang but communicate works
<pre><code>child = subprocess.Popen(command,
                        shell=True,
                        env=environment,
                        close_fds=True,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        stdin=sys.stdin,
                        preexec_fn=os.setsid
                       )

child_interrupted = False
while child.poll() is None:
    if Signal.isInterrupted():
        child_interrupted = True
        os.killpg(os.getpgid(child.pid), signal.SIGTERM)
        break
    time.sleep(0.1)

subout = child.communicate()[0]
logging.info(subout)
</code></pre>

<p>the above works for most command it executes (90%) but for some commands it hangs</p>

<p>for those command that repeatedly hangs, if i get rid of the below, it works fine: </p>

<pre><code>    child_interrupted = False
    while child.poll() is None:
        if Signal.isInterrupted():
            child_interrupted = True
            os.killpg(os.getpgid(child.pid), signal.SIGTERM)
            break
        time.sleep(0.1)
</code></pre>

<p>im assuming for those hanging commands, <code>child.poll() is None</code> even though the job is finished??</p>

<p>communicate() can tell the process is finished but poll() cant?</p>

<p>i've executed <code>ps -ef</code> on those processes<br>
and they are defunct only when <code>child.poll()</code> code is in place<br>
any idea why?</p>

<p>it looks like defunct means "That's a zombie process, it's finished but the parent hasn't wait()ed for it yet."
well, im polling to see if i can call wait/communitcate...</p>

<p>You've set the <code>Popen</code> object to receive the subprocess's <code>stdout</code> via pipe. Problem is, you're not reading from that pipe until the process exits. If the process produces enough output to fill the OS level pipe buffers, and you don't drain the pipe, then you're deadlocked; the subprocess wants you to read the output its writing so it can continue to write, then exit, while you're waiting for it to exit before you'll read the output.</p>

<p>If your explicit poll and interrupt checking is necessary, the easiest solution to this deadlock is probably to launch a thread that drains the pipe:</p>

<pre><code>... launch the thread just after Popen called ...

draineddata = []
# Trivial thread just reads lines from stdout into the list
drainerthread = threading.Thread(target=draineddata.extend, args=(child.stdout,))
drainerthread.daemon = True
drainerthread.start()

... then where you had been doing communicate, change it to: ...
child.wait()
drainerthread.join()
subout = b''.join(draineddata)  # Combine the data read back to a single output
</code></pre>

python subprocess popen