Python's Popen + communicate only returning the first line of stdout
<p>I'm trying to use my command-line git client and Python's I/O redirection in order to automate some common operations on a lot of git repos.
(Yes, this is hack-ish.  I might go back and use a Python library to do this later, but for now it seems to be working out ok :) )</p>

<p>I'd like to be able to capture the output of calling git.  Hiding the output will look nicer, and capturing it will let me log it in case it's useful.</p>

<p><strong>My problem is that I can't get more than the first line of output when I run a 'git clone' command</strong>.  Weirdly, the same code with 'git status' seems to work just fine.</p>

<p>I'm running Python 2.7 on Windows 7 and I'm using the cmd.exe command interpreter.</p>

<p>My sleuthing so far:</p>

<ol>
<li><p>When I call subprocess.call() with "git clone" it runs fine and I
see the output on the console (which confirms that git is producing
output, even though I'm not capturing it). This code:</p>

<pre><code>dir = "E:\\Work\\etc\\etc"
os.chdir(dir)
git_cmd = "git clone git@192.168.56.101:Mike_VonP/bit142_assign_2.git"

#print "SUBPROCESS.CALL" + "="*20
#ret = subprocess.call(git_cmd.split(), shell=True) 
</code></pre>

<p>will produce this output on the console:</p>

<pre><code>SUBPROCESS.CALL====================
Cloning into 'bit142_assign_2'...
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Checking connectivity... done.
</code></pre></li>
<li><p>If I do the same thing with POpen directly, I see the same output on
the console (which is also <em>not</em> being captured). This code:</p>

<pre><code># (the dir = , os.chdir, and git_cmd= lines are still executed here)
print "SUBPROCESS.POPEN" + "="*20
p=subprocess.Popen(git_cmd.split(), shell=True)
p.wait()
</code></pre>

<p>will produce this (effectively identical) output:</p>

<pre><code>SUBPROCESS.POPEN====================
Cloning into 'bit142_assign_2'...
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Checking connectivity... done.
</code></pre>

<p>(Obviously I'm deleting the cloned repo between runs, otherwise I'd
get a 'Everything is up to date' message)</p></li>
<li><p><strong>If I use the communicate() method what I expect is to get a string
that contains all the output that I'm seeing above.  Instead I only
see the line</strong> <code>Cloning into 'bit142_assign_2'...</code>.<br>
This code:</p>

<pre><code>print "SUBPROCESS.POPEN, COMMUNICATE" + "="*20
p=subprocess.Popen(git_cmd.split(), shell=True,\
            bufsize = 1,\
            stderr=subprocess.PIPE,\
            stdout=subprocess.PIPE)
tuple = p.communicate()
p.wait()
print "StdOut:\n" + tuple[0]
print "StdErr:\n" + tuple[1]
</code></pre>

<p>will produce this output:</p>

<pre><code>SUBPROCESS.POPEN, COMMUNICATE====================
StdOut:

StdErr:
Cloning into 'bit142_assign_2'...
</code></pre>

<p>On the one hand I've redirected the output (as you can see from the fact that
it's not in the output) but I'm also only capturing that first line.</p></li>
</ol>

<p>I've tried lots and lots of stuff (calling <code>check_output</code> instead of popen, using pipes with subprocess.call, using pipes with subprocess.popen, and probably other stuff I've forgotten about) but nothing works - I only ever capture that first line of output.</p>

<p>Interestingly, <strong>the exact same code <em>does</em> work correctly with 'git status'</strong>.  Once the repo has been cloned calling git status produces three lines of output (which collectively say 'everything is up to date') and that third example (the POpen+communicate code) does capture all three lines of output.</p>

<p>If anyone has any ideas about what I'm doing wrong or any thoughts on anything I could try in order to better diagnose this problem I would greatly appreciate it.</p>

<p>There are two parts of interest here, one being Python-specific and one being Git-specific.</p>

<h3>Python</h3>

<p>When using the <code>subprocess</code> module, you can elect to control up to three I/O channels of the program you run: stdin, stdout, and stderr.  This is true for <code>subprocess.call</code> and <code>subprocess.check_call</code> as well as <code>subprocess.Popen</code>, but both <code>call</code> and <code>check_call</code> immediately call the new process object's <code>wait</code> method, so for various reasons, it's unwise to supply <code>subprocess.PIPE</code> for the stdout and/or stderr with these two operations.<sup>1</sup></p>

<p>Other than that, using <code>subprocess.call</code> is equivalent to using <code>subprocess.Popen</code>.  In fact, the code for <code>call</code> is a one-liner:</p>

<pre><code>def call(*popenargs, **kwargs):
    return Popen(*popenargs, **kwargs).wait()
</code></pre>

<p>If you choose not to redirect any of the I/O channels, programs that read input get it from the same place Python would, programs that write output to stdout write it to the same place your own Python code would,<sup>2</sup> and programs that write output to stderr write it to the same place Python would.</p>

<p>You can, of course, redirect stdout and/or stderr to actual files, as well as to <code>subprocess.PIPE</code>s.  Files and pipes are <em>not</em> interactive "terminal" or "tty" devices (i.e., are not seen as being directly connected to a human being).  This leads us to Git.</p>

<h3>Git</h3>

<p>Git programs may generally read from stdin and/or write to stdout and/or stderr.  Git may also invoke additional programs, which may do the same, or may bypass these standard I/O channels.</p>

<p>In particular, <code>git clone</code> mainly writes to its stderr, as you have observed.  Moreover, as <a href="http://stackoverflow.com/a/39564562/1256452">mhawke answered</a>, you must add <code>--progress</code> to make Git write progress messages to stderr Git is not talking to an interactive tty device.</p>

<p>If Git needs a password or other authentication when cloning via <code>https</code> or <code>ssh</code>, Git will run an auxiliary program to get this.  These programs, for the most part, <em>bypass</em> stdin entirely (by opening <code>/dev/tty</code> on POSIX systems, or the equivalent on Windows), so as to interact with the user.  How well this will work, or whether it will work at all, in your automated environment is a good question (but again outside the scope of this answer).  But this does bring us back to Python, because ...</p>

<h3>Python</h3>

<p>Besides the <code>subprocess</code> module, there are some external libraries, <a href="https://amoffat.github.io/sh/" rel="nofollow"><code>sh</code></a> and <a href="https://pypi.python.org/pypi/pexpect/" rel="nofollow"><code>pexpect</code></a>, and some facilities built into Python itself <a href="https://docs.python.org/2/library/pty.html" rel="nofollow">via the <code>pty</code> module</a>, that can open a pseudo-tty: an interactive tty device that, instead of being connected directly to a human, is connected to your program.</p>

<p>When using ptys, you can have Git behave identically to when it is talking directly to a humanâin fact, "talking to a human" today is actually done with ptys (or equivalent) anyway, since there are programs running the various windowing systems.  Moreover, programs that ask a human for a password may<sup>3</sup> now interact with your own Python code.  This can be good or bad (or even both), so consider whether you want that to happen.</p>

<hr>

<p><sup>1</sup>Specifically, the point of the <code>communicate</code> method is to manage I/O traffic between the up-to-three streams, if any or all of them are <code>PIPE</code>, without having the subprocess wedge.  Imagine, if you will, a subprocess that prints 64K of text to stdout, then 64K of text to stderr, then another 64K of text to stdout, and then reads from stdin.  If you try to read or write any of these in any specific order, the subprocess will "get stuck" waiting for you to clear something else, while you'll get stuck waiting for the subprocess to finish whichever one you chose to complete first.  What <code>communicate</code> does instead is to use threads or OS-specific non-blocking I/O methods to feed the subprocess input <em>while</em> reading its stdout and stderr, all simultaneously.</p>

<p>In other words, it handled <em>multiplexing</em>.  Thus, if you are not supplying <code>subprocess.PIPE</code> for at least <em>two</em> of the three I/O channels, it's safe to bypass the <code>communicate</code> method.  If you <em>are</em>, it is not (unless you implement your own multiplexing).</p>

<p>There's a somewhat curious edge case here: if you supply <code>subprocess.STDOUT</code> for the stderr output, this tells Python to direct the two outputs of the subprocess into a single communications channel.  This counts as only one pipe, so if you combine the subprocess's stdout and stderr, and supply no input, you can bypass the <code>communicate</code> method.</p>

<p><sup>2</sup>In fact, the subprocess inherits the <em>process</em>'s stdin, stdout, and stderr, which may not match Python's <code>sys.stdin</code>, <code>sys.stdout</code>, and <code>sys.stderr</code> if you've over-ridden those.  This gets into details probably best ignored here. :-)</p>

<p><sup>3</sup>I say "may" instead of "will" because <code>/dev/tty</code> accesses the <em>controlling terminal</em>, and not all ptys are controlling terminals.  This also gets complicated and OS-specific and is also beyond the scope of this answer.</p>

python git popen communicate