How to process Python Pandas data frames in batches?
<p>I have three very long lists of Pandas data frames. For example:</p>

<pre><code>list_a = [tablea1, tablea2, tablea3, tablea4]

list_b = [tableb1, tableb2, tableb3, tableb4]

list_c = [tablec1, tablec2, tablec3, tablec4]
</code></pre>

<p>I want to do something like this:</p>

<pre><code>tablea1 = pd.concat([tablea1, tableb1, tablec1], axis=1)
</code></pre>

<p>So naively, I wrote such codes:</p>

<pre><code>for i in range(len(list_a)):

    list_a[i] = pd.concat([list_a[i], list_b[i], list_c[i]], axis=1)
</code></pre>

<p>This code failed to work, b/c list_a[0] is a reference to tablea1 initially, then inside the loop, list_a[0] will be re-assigned to point to </p>

<pre><code>pd.concat([tablea1, tableb1, tablec1], axis=1), 
</code></pre>

<p>which is a new object. In the end, tablea1 is not modified. (list_a does contain the desired result. But I do want to modify tablea1.) I have spent hours on this and cannot find out a solution. Any help? Thanks.</p>

<p>@qqzj The problem that you will run into is that python doesn't exactly have this feature available. As @Boud <a href="http://stackoverflow.com/a/8989916/624829">mentions</a>, the reference to tablea1, tableb1, tablec1 etc are lost after concatenation. </p>

<p>I'll illustrate a quick and dirty example of the workaround (which is very inefficient, but will get the job done). </p>

<p>Without your data, I'm basically creating random data frames. </p>

<pre><code>tablea1 = pd.DataFrame(np.random.randn(10, 4))
tableb1 = pd.DataFrame(np.random.randn(10, 4))
tablec1 = pd.DataFrame(np.random.randn(10, 4))

tablea2 = pd.DataFrame(np.random.randn(10, 4))
tableb2 = pd.DataFrame(np.random.randn(10, 4))
tablec2 = pd.DataFrame(np.random.randn(10, 4))
</code></pre>

<p>Applying your code to iterate over this list</p>

<pre><code>list_a = [tablea1, tablea2]
list_b = [tableb1, tableb2]
list_c = [tablec1, tablec2]
for i in range(len(list_a)):
    list_a[i] = pd.concat([list_a[i], list_b[i], list_c[i]], axis=1)
</code></pre>

<p>Once you run a compare here, you see the issue that you have highlighted, namely that while <code>list_a[i]</code> has been concatenated with <code>tablea1</code>, <code>tableb1</code>, and <code>tablec1</code>, this hasn't been assigned back to <code>tablea1</code>. </p>

<p>As I mentioned in the comment, the answer is to assign <code>tablea1</code> with the list[0]</p>

<pre><code>tablea1=list_a[0]
</code></pre>

<p>You would repeat this for <code>tablea2</code> <code>tablea3</code> etc. </p>

<p>Doing the compare, you can see now that tablea1 matches the values in list[0]</p>

<pre><code>tablea1==list_a[0]

      0     1     2     3     0     1     2     3     0     1     2     3
0  True  True  True  True  True  True  True  True  True  True  True  True
1  True  True  True  True  True  True  True  True  True  True  True  True
2  True  True  True  True  True  True  True  True  True  True  True  True
3  True  True  True  True  True  True  True  True  True  True  True  True
4  True  True  True  True  True  True  True  True  True  True  True  True
5  True  True  True  True  True  True  True  True  True  True  True  True
6  True  True  True  True  True  True  True  True  True  True  True  True
7  True  True  True  True  True  True  True  True  True  True  True  True
8  True  True  True  True  True  True  True  True  True  True  True  True
9  True  True  True  True  True  True  True  True  True  True  True  True
</code></pre>

<p>Again this is not the ideal solution, but what you are looking for doesn't seem to be the 'pythonic' way. </p>

python pandas