python subprocess.Popen hanging
<pre><code>            child = subprocess.Popen(command,
                         shell=True,
                         env=environment,
                         close_fds=True,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         bufsize=1,
                                     )

            subout = ""
            with child.stdout:
                for line in iter(child.stdout.readline, b''):
                    subout += line
            logging.info(subout)
            rc = child.wait()
</code></pre>

<p>some times (intermittently) this hangs forever.
not sure if it hangs on <code>iter(child.stdout.readline)</code> or <code>child.wait()</code></p>

<p>i <code>ps -ef</code> for the process it Popens and that process no longer exists</p>

<p>my guess is that it has do with bufsize so that child.stdout.readline is going on forever but i have no idea how to test it and as this happens intermittently</p>

<p>I could implement alarm but i m not sure if that's appropriate as i cant really tell whether the popen'd process is just slow or hanging</p>

<p>let's assume that either child.stdout.readline or wait() hangs forever, what actions could i take besides alarm ?</p>

<p>You're likely hitting the deadlock that's <a href="https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait" rel="nofollow">explained in the documentation</a>:</p>

<blockquote>
  <p><code>Popen.wait()</code>:</p>
  
  <p>Wait for child process to terminate. Set and return <code>returncode</code> attribute.</p>
  
  <p><strong>Warning:</strong> This will deadlock when using <code>stdout=PIPE</code> and/or <code>stderr=PIPE</code> and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use <code>communicate()</code> to avoid that.</p>
</blockquote>

<p>The solution is to use <a href="https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate" rel="nofollow"><code>Popen.communicate()</code></a>.</p>

python subprocess