Reading multiple .csv files from different directories into pandas DataFrame
<p>My DataFrame has a index SubjectID, and each Subject ID has its own directory. In each Subject directory is a .csv file with info that I want to put into my DataFrame. Using my SubjectID index, I want to read in the header of the .csv file for every subject and put it into a new column in my DataFrame. </p>

<p>Each subject directory has the same pathway except for the individual subject number.</p>

<p>I have found ways to read multiple .csv files from a single target directory into a pandas DataFrame, but not from multiple directories. Here is some code I have for importing multiple .csv files from a target directory:</p>

<pre><code>subject_path = ('/home/mydirectory/SubjectID/')
filelist = []
os.chdir('subject_path')
for files in glob.glob( "*.csv" ) :
    filelist.append(files)

# read each csv file into single dataframe and add a filename reference column 
df = pd.DataFrame()
columns = range(1,100)
for c, f in enumerate(filelist) :
    key = "file%i" % c
    frame = pd.read_csv( (subject_path + f), skiprows = 1, index_col=0, names=columns )
    frame['key'] = key
    df = df.append(frame,ignore_index=True)
</code></pre>

<p>I want to do something similar but iteratively go into the different Subject directories instead of having a single target directory. </p>

<p>Edit:
I think I want to do this using <code>os</code> not <code>pandas</code>, is there a way to use a loop to search through multiple directories using <code>os</code>?</p>

<p>Assuming your subject folders are in <code>mydirectory</code>, you can just create a list of all folders in the directory and then add the csv's into your filelist.</p>

<pre><code>import os

parent_dir = '/home/mydirectory'
subject_dirs = [os.path.join(parent_dir, dir) for dir in os.listdir(parent_dir) if os.path.isdir(os.path.join(parent_dir, dir))]

filelist = []
for dir in subject_dirs:
    csv_files = [os.path.join(dir, csv) for csv in os.listdir(dir) if os.path.isfile(os.path.join(dir, csv)) and csv.endswith('.csv')]
    for file in csv_files:
        filelist.append(file)

# Do what you did with the dataframe from here
...
</code></pre>

python csv pandas dataframe operating-system