Reading multiple .csv files from different directories into pandas DataFrame
<p>My DataFrame has a index SubjectID, and each Subject ID has its own directory. In each Subject directory is a .csv file with info that I want to put into my DataFrame. Using my SubjectID index, I want to read in the header of the .csv file for every subject and put it into a new column in my DataFrame. </p>

<p>Each subject directory has the same pathway except for the individual subject number.</p>

<p>I have found ways to read multiple .csv files from a single target directory into a pandas DataFrame, but not from multiple directories. Here is some code I have for importing multiple .csv files from a target directory:</p>

<pre><code>subject_path = ('/home/mydirectory/SubjectID/')
filelist = []
os.chdir('subject_path')
for files in glob.glob( "*.csv" ) :
    filelist.append(files)

# read each csv file into single dataframe and add a filename reference column 
df = pd.DataFrame()
columns = range(1,100)
for c, f in enumerate(filelist) :
    key = "file%i" % c
    frame = pd.read_csv( (subject_path + f), skiprows = 1, index_col=0, names=columns )
    frame['key'] = key
    df = df.append(frame,ignore_index=True)
</code></pre>

<p>I want to do something similar but iteratively go into the different Subject directories instead of having a single target directory. </p>

<p>Edit:
I think I want to do this using <code>os</code> not <code>pandas</code>, is there a way to use a loop to search through multiple directories using <code>os</code>?</p>

<p>Consider the recursive method of <a href="https://www.tutorialspoint.com/python/os_walk.htm" rel="nofollow">os.walk()</a> to read all directories and files <em>top-down</em> (default=<code>TRUE</code>) or <em>bottom-up</em>. Additionally, you can use regex to check names to filter specifically for .csv files. </p>

<p>Below will import ALL csv files in any child/grandchild folder from the target root <em>/home/mydirectory</em>. So, be sure to check if non-subject csv files exist, else adjust <code>re.match()</code> accordingly:</p>

<pre><code>import os, re
import pandas as pd

# CURRENT DIRECTORY (PLACE SCRIPT IN /home/mydirectory)
cd = os.path.dirname(os.path.abspath(__file__))

i = 0
columns = range(1,100)
dfList = []

for root, dirs, files in os.walk(cd):
    for fname in files:
        if re.match("^.*.csv$", fname):
            frame = pd.read_csv(os.path.join(root, fname), skiprows = 1, 
                                index_col=0, names=columns)
            frame['key'] = "file{}".format(i)
            dfList.append(frame)    
            i += 1

df = pd.concat(dfList)
</code></pre>

python csv pandas dataframe operating-system