Appending data to Dataframe
<p>I have a directory which contains some csv files called :</p>

<pre><code>results_roll_3_oe_2016-02-04
results_roll_2_oe_2016-01-28
</code></pre>

<p>results_roll_3_oe_2016-02-04 looks like:</p>

<pre><code>date           day_performance
2016-01-26   3.714011839374111
2016-01-27  -8.402334555591418
2016-01-28  -41.09889373400086
</code></pre>

<p>results_roll_2_oe_2016-01-28 looks like:</p>

<pre><code>date           day_performance
2016-02-02   52.07647107113144
2016-02-03    -1.7503249876724
2016-02-04  -158.1667860104882
</code></pre>

<p>(In reality there are more files that this). I am trying to look though the directory sticking together the result_roll csv fies into one dataframe (so my eventual output would look like):</p>

<pre><code>date           day_performance
2016-01-26   3.714011839374111
2016-01-27  -8.402334555591418
2016-01-28  -41.09889373400086
2016-02-02   52.07647107113144
2016-02-03    -1.7503249876724
2016-02-04  -158.1667860104882
</code></pre>

<p>I have written some code (below) that can loop through the files and tries to append the result_roll files together into a new dataframe (<code>dfs</code>), but I get the following output:</p>

<pre><code>date           day_performance
2016-02-02   52.07647107113144
2016-02-03    -1.7503249876724
2016-02-04  -158.1667860104882
date           day_performance
2016-02-02   52.07647107113144
2016-02-03    -1.7503249876724
2016-02-04  -158.1667860104882
</code></pre>

<p>Where it looks to be taking result_roll_2 and appending the data twice along with the header twice.</p>

<p>My code is as follwos:</p>

<pre><code>def main():

    dfs = pd.DataFrame()
    ResultsDataPath = 'C:/Users/stacey/Documents/data/VwapBacktestResults/'
    print(ResultsDataPath)

    allfiles = glob.glob(os.path.join(ResultsDataPath, "*oe*"))
    for fname in allfiles:    
        df = pd.read_csv(fname, header=None, usecols=[1,2], 
                        parse_dates=[0], dayfirst=True,
                        index_col=[0], names=['date', 'day_performance'])
        print(df)

        dfs = df.append(df,ignore_index=False)
</code></pre>

<p>Any help on obtaining my desired output much appreciated.</p>

<p>my exact CSV (results_roll_2_oe_2016-01-28) looks like :</p>

<pre><code>    date    day_performance
0   26/01/2016  3.714011839
1   27/01/2016  -8.402334556
2   28/01/2016  -41.09889373
</code></pre>

<p>and my CSV (results_roll_3_oe_2016-02-04) looks like:</p>

<pre><code>    date    day_performance
0   02/02/2016  52.07647107
1   03/02/2016  -1.750324988
2   04/02/2016  -158.166786
</code></pre>

<p>They are both MS excel Comma Separated Values files</p>

<p>Thanks       </p>

<p><strong>UPDATE:</strong> your CSV files are either space-delimited or TAB-delimited, so you have to specify it. Beside that all your CSV files have a header line and have only two columns, so you don't need to use <code>usecols</code>, <code>header</code>, <code>names</code> parameters:</p>

<pre><code>In [100]: fmask = r'D:\temp\.data\results_roll_*'

In [101]: df = get_merged_csv(glob.glob(fmask),
   .....:                     delim_whitespace=True,
   .....:                     index_col=0)

In [102]: df['date'] = pd.to_datetime(df['date'], dayfirst=True)

In [103]: df
Out[103]:
        date  day_performance
0 2016-01-26         3.714012
1 2016-01-27        -8.402335
2 2016-01-28       -41.098894
3 2016-02-02        52.076471
4 2016-02-03        -1.750325
5 2016-02-04      -158.166786
</code></pre>

<p><strong>OLD answer:</strong></p>

<p>try this:</p>

<pre><code>import glob
import pandas as pd

def get_merged_csv(flist, **kwargs):
    return pd.concat([pd.read_csv(f, **kwargs) for f in flist], ignore_index=True)

fmask = '/path/to/results_roll_*.csv'

df = get_merged_csv(glob.glob(fmask),
                    header=None, usecols=[1,2], 
                    parse_dates=[0], dayfirst=True,
                    index_col=[0], names=['date', 'day_performance'])
</code></pre>

python pandas