Get output of pv using python
<p>Is there a way to use the program pv from within python so as to get the progress of an operation?</p>

<p>So far I have the following:</p>

<pre><code>    p0 = sp.Popen(["pv", "-f", args.filepath],
                  bufsize=0,
                  stdout=sp.PIPE,
                  stderr=sp.PIPE)
    p1 = sp.Popen(["awk", "{print $1, $2, $1, $3, $4 }",  "{}".format(args.filepath)],
                   stdout=sp.PIPE,
                   stdin=p0.stdout)
</code></pre>

<p>But I'm having trouble getting continuous output from <code>p0</code>. I tried:</p>

<pre><code>    for line in p0.stderr:
        print("line:", line)
</code></pre>

<p>But this waits for the process to finish and then only prints the last progress report from <code>pv</code>. Does anybody know how I can get it to print the continuously updating status?</p>

<p>It turns out <code>pv</code> outputs each line with a carriage return at end (<code>\r</code>). To be able to continuously read from the output, <code>Popen</code> needs to be initialized with <code>universal_lines=True</code>, like this:</p>

<pre><code>    p0 = sp.Popen(['pv', '-f', args.filepath],
                    stdout=sp.PIPE,
                    stderr=sp.PIPE,
                    universal_newlines=True)
</code></pre>

<p>This leads to a continuous output of progress reports:</p>

<pre><code>line: 7.12MB 0:00:01 [ 7.1MB/s] [=&gt;                                  ]  8% ETA 0:00:11

line: 14.6MB 0:00:02 [7.42MB/s] [====&gt;                               ] 16% ETA 0:00:10

line: 22.1MB 0:00:03 [7.55MB/s] [=======&gt;                            ] 24% ETA 0:00:09

line: 29.5MB 0:00:04 [7.36MB/s] [==========&gt;                         ] 33% ETA 0:00:08
</code></pre>

<p>Here's a reference to a similar question:</p>

<p><a href="http://stackoverflow.com/questions/4620547/real-time-output-of-subprocess-popen-and-not-line-by-line">Real time output of subprocess.popen() and not line by line</a></p>

python subprocess