Python Iterating Through Large Data Set and Deleting Assessed Data
<p>I am working with a data set with 10,000 customers data from months 1-12. I am generating correlations for different values over the 12 month period for each customer.</p>

<p>Currently my output correlation file has more rows than my original file. I realize this is an iteration error from when I am trying to delete the already assessed rows from the original data set.</p>

<p>The result I expect is a data set of 10,000 entries of various correlations corresponding to each customers yearly assessment.</p>

<p>I have bolded (starred) where I believe the error is.</p>

<p>Here is my current code:</p>

<pre><code>for x_customer in range(0,len(overalldata),12):

        for x in range(0,13,1):
                cust_months = overalldata[0:x,1]

                cust_balancenormal = overalldata[0:x,16]

                cust_demo_one = overalldata[0:x,2]
                cust_demo_two = overalldata[0:x,3]

                num_acct_A = overalldata[0:x,4]
                num_acct_B = overalldata[0:x,5]

                out_mark_channel_one = overalldata[0:x,25]
                out_service_channel_two = overalldata[0:x,26]
                out_mark_channel_three = overalldata[0:x,27]
                out_mark_channel_four = overalldata[0:x,28]


    #Correlation Calculations

                #Demographic to Balance Correlations
                demo_one_corr_balance = numpy.corrcoef(cust_balancenormal, cust_demo_one)[1,0]
                demo_two_corr_balance = numpy.corrcoef(cust_balancenormal, cust_demo_two)[1,0]


                #Demographic to Account Number Correlations
                demo_one_corr_acct_a = numpy.corrcoef(num_acct_A, cust_demo_one)[1,0]
                demo_one_corr_acct_b = numpy.corrcoef(num_acct_B, cust_demo_one)[1,0]
                demo_two_corr_acct_a = numpy.corrcoef(num_acct_A, cust_demo_two)[1,0]
                demo_two_corr_acct_b = numpy.corrcoef(num_acct_B, cust_demo_two)[1,0]

                #Marketing Response Channel One
                mark_one_corr_acct_a = numpy.corrcoef(num_acct_A, out_mark_channel_one)[1, 0]
                mark_one_corr_acct_b = numpy.corrcoef(num_acct_B, out_mark_channel_one)[1, 0]
                mark_one_corr_balance = numpy.corrcoef(cust_balancenormal, out_mark_channel_one)[1, 0]

                #Marketing Response Channel Two
                mark_two_corr_acct_a = numpy.corrcoef(num_acct_A, out_service_channel_two)[1, 0]
                mark_two_corr_acct_b = numpy.corrcoef(num_acct_B, out_service_channel_two)[1, 0]
                mark_two_corr_balance = numpy.corrcoef(cust_balancenormal, out_service_channel_two)[1, 0]

                #Marketing Response Channel Three
                mark_three_corr_acct_a = numpy.corrcoef(num_acct_A, out_mark_channel_three)[1, 0]
                mark_three_corr_acct_b = numpy.corrcoef(num_acct_B, out_mark_channel_three)[1, 0]
                mark_three_corr_balance = numpy.corrcoef(cust_balancenormal, out_mark_channel_three)[1, 0]

                #Marketing Response Channel Four
                mark_four_corr_acct_a = numpy.corrcoef(num_acct_A, out_mark_channel_four)[1, 0]
                mark_four_corr_acct_b = numpy.corrcoef(num_acct_B, out_mark_channel_four)[1, 0]
                mark_four_corr_balance = numpy.corrcoef(cust_balancenormal, out_mark_channel_four)[1, 0]


                #Result Correlations For Exporting to CSV of all Correlations
                result_correlation = [(demo_one_corr_balance),(demo_two_corr_balance),(demo_one_corr_acct_a),(demo_one_corr_acct_b),(demo_two_corr_acct_a),(demo_two_corr_acct_b),(mark_one_corr_acct_a),(mark_one_corr_acct_b),(mark_one_corr_balance),
                                      (mark_two_corr_acct_a),(mark_two_corr_acct_b),(mark_two_corr_balance),(mark_three_corr_acct_a),(mark_three_corr_acct_b),(mark_three_corr_balance),(mark_four_corr_acct_a),(mark_four_corr_acct_b),
                                      (mark_four_corr_balance)]
                result_correlation_nan_nuetralized = numpy.nan_to_num(result_correlation)
                c.writerow(result_correlation)

        **result_correlation_combined = emptylist.append([result_correlation])
        cust_delete_list = [0,x_customer,1]
        overalldata = numpy.delete(overalldata, (cust_delete_list), axis=0)**
</code></pre>

<p>This may not completely solve your problem, but I think it's relevant.</p>

<p>When you run <code>.append</code> on a list object (empty or otherwise), the value returned by that method is <code>None</code>. So, with the line <code>result_correlation_combined = emptylist.append([result_correlation])</code>, regardless of whether <code>empty_list</code> is an empty or non-empty list, the value of <code>result_correlation_combined</code> will be <code>None</code>.</p>

<p>Here's a simple example of what I'm talking about - I'll just make up some numbers since no data were provided.</p>

<pre><code>&gt;&gt;&gt; empty_list = []
&gt;&gt;&gt; result_correlation = []

&gt;&gt;&gt; for j in range(10):
        result_correlation.append(j)

&gt;&gt;&gt; result_correlation
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

&gt;&gt;&gt; result_correlation_combined = empty_list.append(result_correlation)
&gt;&gt;&gt; print(result_correlation_combined)
None
</code></pre>

<p>So, you could run <code>result_correlation_combined.append(result_correlation)</code> or <code>result_correlation_combined += result_correlation</code>, or even <code>result_correlation_combined.extend(result_correlation)</code>... They will all produce the same result. See if that gives you the answer you're looking for. If not, come back.</p>

python list csv for-loop iteration