Concatinating multiple Data frames of different length
<p>I have 88 different dataFrame of different lengths, which I need to concatenate. And its all are located in one directory and I used the following python script to produce such a single data frame.</p>

<p>Here is what I tried,</p>

<pre><code> path = 'GTFS/' 
    files = os.listdir(path)

    files_txt  = [os.path.join(path,i) for i in files if i.endswith('.tsv')]

    ## Change it into dataframe
    dfs = [pd.DataFrame.from_csv(x, sep='\t')[[6]] for x in files_txt]
    ##Concatenate it
    merged = pd.concat(dfs,axis=1)
</code></pre>

<p>Since each of those data frames are of different length or shape its throwing me following error message,</p>

<pre><code>ValueError: Shape of passed values is (88, 57914), indices imply (88, 57905)
</code></pre>

<p>My aim is to concatenate column-wise into single data frame with 88 columns, as my input is 88 separate data frames from which I need to use 7th column as in my script.
Any solutions or suggestion would be great in this case for concatenating data frames
Thank you</p>

<p>The key is to make a <code>list</code> of different data-frames and then concatenate the list instead of individual concatenation.</p>

<p>I created 10 <code>df</code> filled with random length data of one column and saved to <code>csv</code> files to simulate your data.</p>

<pre><code>import pandas as pd
import numpy as np
from random import randint


#generate 10 df and save to seperate csv files
for i in range(1,11):
    dfi = pd.DataFrame({'a':np.arange(randint(2,11))})
    csv_file = "file{0}.csv".format(i)
    dfi.to_csv(csv_file, sep='\t')
    print "saving file", csv_file
</code></pre>

<p>Then we read those 10 <code>csv</code> files into separate data-frames and save to a <code>list</code></p>

<pre><code>#read previously saved csv files into 10 seperate df
# and add to list
frames = []
for x in range(1,10):
    csv_file = "file{0}.csv".format(x)
    newdf = pd.DataFrame.from_csv(csv_file,  sep='\t')
    frames.append(newdf)
</code></pre>

<p>Finally, we concatenate the <code>list</code></p>

<pre><code>#concatenate frames list
result = pd.concat(frames, axis=1)
print result
</code></pre>

<p>The result is 10 frames of variable length concatenated column wise into single <code>df</code>.</p>

<pre><code>saving file file1.csv
saving file file2.csv
saving file file3.csv
saving file file4.csv
saving file file5.csv
saving file file6.csv
saving file file7.csv
saving file file8.csv
saving file file9.csv
saving file file10.csv
      a    a    a    a    a    a    a   a    a
0   0.0  0.0  0.0  0.0  0.0  0.0  0.0   0  0.0
1   1.0  1.0  1.0  1.0  1.0  1.0  1.0   1  1.0
2   2.0  2.0  2.0  2.0  2.0  2.0  2.0   2  2.0
3   3.0  3.0  3.0  3.0  3.0  NaN  3.0   3  NaN
4   4.0  4.0  4.0  4.0  4.0  NaN  NaN   4  NaN
5   5.0  5.0  5.0  5.0  5.0  NaN  NaN   5  NaN
6   6.0  6.0  6.0  6.0  6.0  NaN  NaN   6  NaN
7   NaN  7.0  7.0  7.0  7.0  NaN  NaN   7  NaN
8   NaN  8.0  NaN  NaN  8.0  NaN  NaN   8  NaN
9   NaN  NaN  NaN  NaN  9.0  NaN  NaN   9  NaN
10  NaN  NaN  NaN  NaN  NaN  NaN  NaN  10  NaN
</code></pre>

<p>Hope this is what you are looking for. A good example on merge, join and concatenate can be found <a href="http://pandas.pydata.org/pandas-docs/stable/merging.html" rel="nofollow">here</a>.</p>

python pandas numpy dataframe