Python: read files from directory and concatenate that
<p>I have a folder <code>et</code> with <code>.csv</code> files and I try to read that and next concatenate that and get one file.
I try </p>

<pre><code>import os

path = 'et/'
for filename in os.listdir(path):
    et = open(filename)
    print et
</code></pre>

<p>but I get an error </p>

<pre><code>Traceback (most recent call last):
File "C:/Users/ï¿½ï¿½ï¿½ï¿½ï¿½ ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½/Desktop/projects/PMI/join et.py", line 5, in &lt;module&gt;
et = open(filename)
IOError: [Errno 2] No such file or directory: '0et.csv'
</code></pre>

<p>I can't understand, why I get this error, because when I 
<code>print filename</code>
I get</p>

<pre><code>0et.csv
1et.csv
2et.csv
3et.csv
4et.csv
5et.csv
6et.csv
7et.csv
8et.csv
</code></pre>

<p>Using glob.glob will be a better option, along with using os.path.join to get to the full path:</p>

<pre><code>from glob import glob
from os.path import join, abspath
from os import listdir, getcwd

import pandas as pd

data_frame = pd.DataFrame()
dir_path = "et"
full_path = join(abspath(getcwd()), dir_path, "*.csv")
for file_name in glob(full_path):
    csv_reader = pd.read_csv(file_name, names=columns)
    # Guessing that all csv files will have the header
    #If header is absent, use names=None
    data_frame = data_frame.append(csv_reader, ignore_index=True)
    # There is also a concat funtion to use. I am comfortable with append
    # For concat, it will be data_frame = pd.concat(data_frame, csv_reader, ignore_index=True)
</code></pre>

<ol>
<li>abspath will make sure that the full directory from the root(in case of windows, from the main file system drive) is taken</li>
<li>Adding *.csv with the join will make sure that you will check for csv files with the directory</li>
<li>glob(full_path) will return a list of csv files, with absolute path, of the given directory</li>
<li>Always make sure that you will either close the file descriptor explicitly or use the with statement to do it automatically, as it is a clean practice. Any C developer can vouch that closing the file descriptor is best. Since we need to put the value in the dataframe, I took out the with statement and added the read_csv from pandas.</li>
<li>pandas.read_csv will make life lot better while reading the csv, in case we are into writing the csv file contents to dataframes. With read_csv and pandas append(or concat), we can write csv files easily without writing the header content from other csv files. I have given in append, because of personal opinion. Added how to use concat in the comment though.</li>
</ol>

python pandas directory