How to use subprocess and 'cat' to read in data line by line?
<p>I'm having trouble understanding how to use <code>subprocess</code> for a problem on mine. </p>

<p>Let's say I have a tab-delimited text file <code>tabdelimited1.txt</code> in my subdiretory which I would like to read into a pandas dataframe. </p>

<p>Naturally, you could simply import the data as follows: </p>

<pre><code>import pandas as pd
df = pd.read_csv("tabdelimited1.txt", header=None, sep="\s+")
</code></pre>

<p>However, let's say we wanted to use <code>subprocess</code>. In the command line, <code>$cat tabdelimited1.txt</code> will output all of the lines. </p>

<p>Now, I want to use subprocess to read the output of <code>cat tabdelimited1.txt</code>. How does one do this? </p>

<p>We could use </p>

<pre><code>import subprocess
task = subprocess.Popen("cat file.txt", shell=True,  stdout=subprocess.PIPE)
data = task.stdout.read()
</code></pre>

<p>but (1) I get an error for <code>shell=True</code> and (2) I would like to read in the data line-by-line. </p>

<p>How can I use <code>subprocess</code> to read <code>tabdelimited1.txt</code> line-by-line? The script should look something like this:</p>

<pre><code>import subprocess
import pandas as pd

df = pd.DataFrame()
task = subprocess.Popen("cat file.txt", shell=True,  stdout=subprocess.PIPE)
# while lines exist:
    # line = subprocess std
    df=pd.concat([df, line])
</code></pre>

<p>EDITED</p>

<p>You can skip the shell completely by breaking the command into a list. Then its just a matter of iterating the process stdout:</p>

<pre><code>import subprocess
import pandas as pd

df = pd.DataFrame()
task = subprocess.Popen(["cat", "file.txt"], stdout=subprocess.PIPE)
for line in task.stdout:
    df=pd.concat([df, line])
task.wait()
</code></pre>

python python-3.x csv subprocess cat