Python's Popen + communicate only returning the first line of stdout
<p>I'm trying to use my command-line git client and Python's I/O redirection in order to automate some common operations on a lot of git repos.
(Yes, this is hack-ish.  I might go back and use a Python library to do this later, but for now it seems to be working out ok :) )</p>

<p>I'd like to be able to capture the output of calling git.  Hiding the output will look nicer, and capturing it will let me log it in case it's useful.</p>

<p><strong>My problem is that I can't get more than the first line of output when I run a 'git clone' command</strong>.  Weirdly, the same code with 'git status' seems to work just fine.</p>

<p>I'm running Python 2.7 on Windows 7 and I'm using the cmd.exe command interpreter.</p>

<p>My sleuthing so far:</p>

<ol>
<li><p>When I call subprocess.call() with "git clone" it runs fine and I
see the output on the console (which confirms that git is producing
output, even though I'm not capturing it). This code:</p>

<pre><code>dir = "E:\\Work\\etc\\etc"
os.chdir(dir)
git_cmd = "git clone git@192.168.56.101:Mike_VonP/bit142_assign_2.git"

#print "SUBPROCESS.CALL" + "="*20
#ret = subprocess.call(git_cmd.split(), shell=True) 
</code></pre>

<p>will produce this output on the console:</p>

<pre><code>SUBPROCESS.CALL====================
Cloning into 'bit142_assign_2'...
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Checking connectivity... done.
</code></pre></li>
<li><p>If I do the same thing with POpen directly, I see the same output on
the console (which is also <em>not</em> being captured). This code:</p>

<pre><code># (the dir = , os.chdir, and git_cmd= lines are still executed here)
print "SUBPROCESS.POPEN" + "="*20
p=subprocess.Popen(git_cmd.split(), shell=True)
p.wait()
</code></pre>

<p>will produce this (effectively identical) output:</p>

<pre><code>SUBPROCESS.POPEN====================
Cloning into 'bit142_assign_2'...
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (9/9), done.
Checking connectivity... done.
</code></pre>

<p>(Obviously I'm deleting the cloned repo between runs, otherwise I'd
get a 'Everything is up to date' message)</p></li>
<li><p><strong>If I use the communicate() method what I expect is to get a string
that contains all the output that I'm seeing above.  Instead I only
see the line</strong> <code>Cloning into 'bit142_assign_2'...</code>.<br>
This code:</p>

<pre><code>print "SUBPROCESS.POPEN, COMMUNICATE" + "="*20
p=subprocess.Popen(git_cmd.split(), shell=True,\
            bufsize = 1,\
            stderr=subprocess.PIPE,\
            stdout=subprocess.PIPE)
tuple = p.communicate()
p.wait()
print "StdOut:\n" + tuple[0]
print "StdErr:\n" + tuple[1]
</code></pre>

<p>will produce this output:</p>

<pre><code>SUBPROCESS.POPEN, COMMUNICATE====================
StdOut:

StdErr:
Cloning into 'bit142_assign_2'...
</code></pre>

<p>On the one hand I've redirected the output (as you can see from the fact that
it's not in the output) but I'm also only capturing that first line.</p></li>
</ol>

<p>I've tried lots and lots of stuff (calling <code>check_output</code> instead of popen, using pipes with subprocess.call, using pipes with subprocess.popen, and probably other stuff I've forgotten about) but nothing works - I only ever capture that first line of output.</p>

<p>Interestingly, <strong>the exact same code <em>does</em> work correctly with 'git status'</strong>.  Once the repo has been cloned calling git status produces three lines of output (which collectively say 'everything is up to date') and that third example (the POpen+communicate code) does capture all three lines of output.</p>

<p>If anyone has any ideas about what I'm doing wrong or any thoughts on anything I could try in order to better diagnose this problem I would greatly appreciate it.</p>

<p>Try adding the <code>--progress</code> option to your git command. This forces git to emit the progress status to stderr even when the the git process is not attached to a terminal - which is the case when running git via the <code>subprocess</code> functions.</p>

<pre><code>git_cmd = "git clone --progress git@192.168.56.101:Mike_VonP/bit142_assign_2.git"

print "SUBPROCESS.POPEN, COMMUNICATE" + "="*20
p = subprocess.Popen(git_cmd.split(), stderr=subprocess.PIPE, stdout=subprocess.PIPE)
tuple = p.communicate()
p.wait()
print "StdOut:\n" + tuple[0]
print "StdErr:\n" + tuple[1]
</code></pre>

<p>N.B. I am unable to test this on Windows, but it is effective on Linux.</p>

<p>Also, it should not be necessary to specify <code>shell=True</code> and this might be a security problem, so it's best avoided.</p>

python git popen communicate