Extracting data from multiple files with python
<p>I'm trying to extract data from a directory with 12 .txt files. Each file contains 3 columns of data (X,Y,Z) that i want to extract. I want to collect all the data in one df(InforDF), but so far i only succeeded in creating a df with all of the X,Y and Z data in the same column. This is my code: </p>

<pre><code>import pandas as pd
import numpy as np
import os
import fnmatch

path = os.getcwd()

file_list = os.listdir(path)

InfoDF = pd.DataFrame()

for file in file_list:
    try:
        if fnmatch.fnmatch(file, '*.txt'):
            filedata = open(file, 'r')
            df = pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'})

    except Exception as e:
        print(e)
</code></pre>

<p>What am i doing wrong? </p>

<ul>
<li>As camilleri mentions above, you are overwriting <code>df</code> in your loop</li>
<li>Also there is no point catching a general exception</li>
</ul>

<p><strong>Solution</strong>: Create an empty dataframe <code>InfoDF</code> before the loop and then use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.append.html" rel="nofollow">append</a> or <a href="http://pandas.pydata.org/pandas-docs/stable/merging.html" rel="nofollow">concat</a> to populate it with smaller <code>df</code>s</p>

<pre><code>import pandas as pd
import numpy as np
import os
import fnmatch

path = os.getcwd()

file_list = os.listdir(path)

InfoDF = pd.DataFrame(columns={'X','Y','Z'}) # create empty dataframe
for file in file_list:
    if fnmatch.fnmatch(file, '*.txt'): 
        filedata = open(file, 'r')
        df = pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'})
        InfoDF.append(df, ignore_index=True)
print InfoDF
</code></pre>

python numpy dataframe