Pandas - make a large DataFrame into several small DataFrames and run each through a function
<p>I have a huge dataset with about 60000 data. I would first use some criteria to do groupby on the whole dataset, and what I want to do next is to separate the whole dataset to many small datasets within the criteria and to run a function to each of the small dataset automatically to get a parameter for each small dataset. I have no idea on how to do this. Is there any code to make it possible?
This is what I have</p>

<pre><code>Date         name    number
20100101    John      1
20100102    Kate      3
20100102    Kate      2
20100103    John      3
20100104    John      1
</code></pre>

<p>And I want it to be split into two small ones</p>

<pre><code>Date         name    number
20100101    John      1
20100103    John      3
20100104    John      1

Date         name    number
20100102    Kate      3
20100102    Kate      2
</code></pre>

<p>I think a more efficient way than filtering the original data set using subsetting is <code>groupby()</code>, as a demo:</p>

<pre><code>for _, g in df.groupby('name'):
    print(g)

#       Date  name  number
#0  20100101  John       1
#3  20100103  John       3
#4  20100104  John       1
#       Date  name  number
#1  20100102  Kate       3
#2  20100102  Kate       2
</code></pre>

<p>So to get a list of small data frames, you can do <code>[g for _, g in df.groupby('name')]</code>.</p>

<p>To expand on this answer, we can see more clearly what <code>df.groupby()</code> returns as follows:</p>

<pre><code>for k, g in df.groupby('name'):
    print(k)
    print(g)

# John
#        Date  name  number
# 0  20100101  John       1
# 3  20100103  John       3
# 4  20100104  John       1
# Kate
#        Date  name  number
# 1  20100102  Kate       3
# 2  20100102  Kate       2
</code></pre>

<p>For each element returned by <code>groupby()</code>, it contains a key and a data frame with <code>name</code> which has a unique value of the key. In the above solution, we don't need the key, so we can just specify a position holder and discard it.</p>

python pandas