How to use subprocess and 'cat' to read in data line by line?
<p>I'm having trouble understanding how to use <code>subprocess</code> for a problem on mine. </p>

<p>Let's say I have a tab-delimited text file <code>tabdelimited1.txt</code> in my subdiretory which I would like to read into a pandas dataframe. </p>

<p>Naturally, you could simply import the data as follows: </p>

<pre><code>import pandas as pd
df = pd.read_csv("tabdelimited1.txt", header=None, sep="\s+")
</code></pre>

<p>However, let's say we wanted to use <code>subprocess</code>. In the command line, <code>$cat tabdelimited1.txt</code> will output all of the lines. </p>

<p>Now, I want to use subprocess to read the output of <code>cat tabdelimited1.txt</code>. How does one do this? </p>

<p>We could use </p>

<pre><code>import subprocess
task = subprocess.Popen("cat file.txt", shell=True,  stdout=subprocess.PIPE)
data = task.stdout.read()
</code></pre>

<p>but (1) I get an error for <code>shell=True</code> and (2) I would like to read in the data line-by-line. </p>

<p>How can I use <code>subprocess</code> to read <code>tabdelimited1.txt</code> line-by-line? The script should look something like this:</p>

<pre><code>import subprocess
import pandas as pd

df = pd.DataFrame()
task = subprocess.Popen("cat file.txt", shell=True,  stdout=subprocess.PIPE)
# while lines exist:
    # line = subprocess std
    df=pd.concat([df, line])
</code></pre>

<p>EDITED</p>

<pre><code>import sys
for line in sys.stdin:
    print(line.split())
</code></pre>

<p>can be used with a shell command like:</p>

<pre><code>0025:~/mypy$ cat x.txt | python3 stack39864304.py
['1', '3', 'test1;']
['2', '2', 'test2;']
['3', '2', 'test3;']
</code></pre>

<p>Otherwise in an interactive session I can do:</p>

<pre><code>In [269]: task = subprocess.Popen("cat x.txt", shell=True,  stdout=subprocess.PIPE)
In [270]: for line in task1.stdout:print(line.split())
[b'1', b'3', b'test1;']
[b'2', b'2', b'test2;']
[b'3', b'2', b'test3;']
</code></pre>

<p>(py3 bytestrings)</p>

<p><code>python3 stack39864304.py &lt; x.txt</code> is another way of sending this file to the script.</p>

<p><code>cat afile | ...</code> is perhaps too simple, and raise all the objections about why not read directly.  But <code>cat</code> can be replaced by <code>head</code>, <code>tail</code> or even <code>ls -l | python3 stack39864304.py</code> to get a directory list with this <code>split</code>.</p>

<p>I use <code>ipython</code> for most of my interactive python coding; many of its <code>%magic</code> use subprocesses;  I use <code>cat x.txt</code>, <code>ls</code> all the time from within this session.</p>

python python-3.x csv subprocess cat