Pandas - make a large DataFrame into several small DataFrames and run each through a function
<p>I have a huge dataset with about 60000 data. I would first use some criteria to do groupby on the whole dataset, and what I want to do next is to separate the whole dataset to many small datasets within the criteria and to run a function to each of the small dataset automatically to get a parameter for each small dataset. I have no idea on how to do this. Is there any code to make it possible?
This is what I have</p>

<pre><code>Date         name    number
20100101    John      1
20100102    Kate      3
20100102    Kate      2
20100103    John      3
20100104    John      1
</code></pre>

<p>And I want it to be split into two small ones</p>

<pre><code>Date         name    number
20100101    John      1
20100103    John      3
20100104    John      1

Date         name    number
20100102    Kate      3
20100102    Kate      2
</code></pre>

<p>Unless your function is really slow, this can probably be accomplished by slicing (e.g. <code>df_small = df[a:b]</code> for some indices a and b). The only trick is to choose a and b. I use <code>range</code> in the code below but you could do it other ways:</p>

<pre><code>param_list = []
n = 10000 #size of smaller dataframe

# loop up to 60000-n, n at a time
for i in range(0,60000-n,n):
    # take a slice of big dataframe and apply function to get 'param'
    df_small = df[i:i+n] #
    param = function( df_small )

    # keep our results in a list
    param_list.append(param)
</code></pre>

<p>EDIT: Based on update, you could do something like this:</p>

<pre><code># loop through names
for i in df.name.values.unique():
    # take a slice of big dataframe and apply function to get 'param'
    df_small = df[df.name==i] 
</code></pre>

python pandas