Extracting data from multiple files with python
<p>I'm trying to extract data from a directory with 12 .txt files. Each file contains 3 columns of data (X,Y,Z) that i want to extract. I want to collect all the data in one df(InforDF), but so far i only succeeded in creating a df with all of the X,Y and Z data in the same column. This is my code: </p>

<pre><code>import pandas as pd
import numpy as np
import os
import fnmatch

path = os.getcwd()

file_list = os.listdir(path)

InfoDF = pd.DataFrame()

for file in file_list:
    try:
        if fnmatch.fnmatch(file, '*.txt'):
            filedata = open(file, 'r')
            df = pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'})

    except Exception as e:
        print(e)
</code></pre>

<p>What am i doing wrong? </p>

<pre><code>df = pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'})
</code></pre>

<p>this line replace <code>df</code> at each iteration of the loop, that's why you only have the last one at the end of your program.</p>

<p>what you can do is to save all your dataframe in a list and concatenate them at the end</p>

<pre><code>df_list = []
for file in file_list:
    try:
        if fnmatch.fnmatch(file, '*.txt'): 
            filedata = open(file, 'r')
            df_list.append(pd.read_table(filedata, delim_whitespace=True, names={'X','Y','Z'}))
df = pd.concat(df_list)
</code></pre>

<p>alternatively, you can write it:</p>

<pre><code>df_list = pd.concat([pd.read_table(open(file, 'r'), delim_whitespace=True, names={'X','Y','Z'})  for file in file_list if fnmatch.fnmatch(file, '*.txt')])
</code></pre>

python numpy dataframe