How to process Python Pandas data frames in batches?
<p>I have three very long lists of Pandas data frames. For example:</p>

<pre><code>list_a = [tablea1, tablea2, tablea3, tablea4]

list_b = [tableb1, tableb2, tableb3, tableb4]

list_c = [tablec1, tablec2, tablec3, tablec4]
</code></pre>

<p>I want to do something like this:</p>

<pre><code>tablea1 = pd.concat([tablea1, tableb1, tablec1], axis=1)
</code></pre>

<p>So naively, I wrote such codes:</p>

<pre><code>for i in range(len(list_a)):

    list_a[i] = pd.concat([list_a[i], list_b[i], list_c[i]], axis=1)
</code></pre>

<p>This code failed to work, b/c list_a[0] is a reference to tablea1 initially, then inside the loop, list_a[0] will be re-assigned to point to </p>

<pre><code>pd.concat([tablea1, tableb1, tablec1], axis=1), 
</code></pre>

<p>which is a new object. In the end, tablea1 is not modified. (list_a does contain the desired result. But I do want to modify tablea1.) I have spent hours on this and cannot find out a solution. Any help? Thanks.</p>

<p>Thanks for zhqiat's sample codes. Let me expand a bit on it. Here this problem can be solved using exec statement.</p>

<pre><code>import pandas as pd
import numpy as np

tablea1 = pd.DataFrame(np.random.randn(10, 4))
tableb1 = pd.DataFrame(np.random.randn(10, 4))
tablec1 = pd.DataFrame(np.random.randn(10, 4))

tablea2 = pd.DataFrame(np.random.randn(10, 4))
tableb2 = pd.DataFrame(np.random.randn(10, 4))
tablec2 = pd.DataFrame(np.random.randn(10, 4))

list_a = [tablea1, tablea2]
list_b = [tableb1, tableb2]
list_c = [tablec1, tablec2]

for i in range(1, len(list_a)+1):
    exec 'tablea' + str(i) + ' = pd.concat([tablea' + str(i) + ', ' + 'tableb' + str(i) + ', ' +  'tablec' + str(i) + '], axis=1)'

print tablea1
</code></pre>

<p>I have been using this approach for a while. But after the code got more complicated. exec started complaining </p>

<pre><code>'SyntaxError: unqualified exec is not allowed in function 'function name' it contains a nested function with free variables'. 
</code></pre>

<p>Here is the troubled codes:</p>

<pre><code>def overall_function():

    def dummy_function():
        return True

    tablea1 = pd.DataFrame(np.random.randn(10, 4))
    tableb1 = pd.DataFrame(np.random.randn(10, 4))
    tablec1 = pd.DataFrame(np.random.randn(10, 4))

    tablea2 = pd.DataFrame(np.random.randn(10, 4))
    tableb2 = pd.DataFrame(np.random.randn(10, 4))
    tablec2 = pd.DataFrame(np.random.randn(10, 4))

    list_a = ['tablea1', 'tablea2']
    list_b = ['tableb1', 'tableb2']
    list_c = ['tablec1', 'tablec2']

    for i, j, k in zip(list_a, list_b, list_c):
        exec(i + ' = pd.concat([' + i + ',' + j + ',' + k + '], axis=1)')


    print tablea1

overall_function()
</code></pre>

<p>This code will generate the error message. The funny thing is that there is no other 'def' statement in my real function at all. So I have no nested function. I am very puzzled why I got such an error message. My question is whether there is a way to ask Python telling me which variable is the culprit, i.e. the free variable that cause the problem? Or, which sub function is the responsible for the failure of my code. Ideally, for this example, I wish I could force python to tell me that dummy_function is the cause.</p>

python pandas