How to efficiently preform loop statements for data analysis?
<p>I wrote this code recently to comb over data points and it works beautifully for small data sets. However whenever the data sets get too big all I get is junk output and a message in the pycharm console that reads <code>! Too much output to process</code>. Then the maximum line output seems to be around 55,000 lines. </p>

<p>The point of this code is to analyze the proximity of coordinates in file1 to all elements in file2. Then return any resulting matches of coordinates which share proximity. As you will see below I wrote a nested for loop to do this which I understand may be a sort of brute force tactic so that could perhaps be the issue in getting the error message later on?</p>

<p>Heres the code:</p>

<pre><code>import numpy as np
import math as ma

filename1 = "C:\Users\Justin\Desktop\file1.data"
data1 = np.genfromtxt(filename1,
                     skip_header=1,
                     usecols=(0, 1))
                     #dtype=[
                            #("x1", "f9"),
                         #("y1", "f9")])
#print "data1", data1

filename2 = "C:\Users\Justin\Desktop\file2.data"
data2 = np.genfromtxt(filename2,
                      skip_header=1,
                      usecols=(0, 1))
                      #dtype=[
                             #("x2", "f9"),
                             #("y2", "f9")])

#print "data2",data2

def d(a,b):
    d = ma.acos(ma.sin(ma.radians(a[1]))*ma.sin(ma.radians(b[1]))
                +ma.cos(ma.radians(a[1]))*ma.cos(ma.radians(b[1]))*       (ma.cos(ma.radians((a[0]-b[0])))))
    return d

results = open("results.txt", "w")

for coor1 in data1:
    for coor2 in data2:
        n=0
        a = [coor1[0], coor1[1]]
        b = [coor2[0], coor2[1]]
        #print "a", a
        #print "b", b

    if d(a, b) &lt; 0.07865:  # if true what happens
        results.write("\t".join([str(coor1), str(coor2), "True", str(d)]) + "\n")

    else:
        results.write("\t".join([str(coor1), str(coor2), "False", str(d)]) + "\n")
results.close()
</code></pre>

<p>Ideally I wont get this issue when I start cross checking data files of over 500,000 coordinates each since I doubt many of them will share much proximity.</p>

<p>But there are 2 reasons for posting this (again sort of). First of all to share this code with anybody who can use it since it has already proved to be a powerful tool to analyze data or coordinates in any arbitrary spherical space. Secondly, to see if anybody had any advice on how to possibly make this more efficient and help me solve the error message?</p>

<p>Most specifically the error message appears when my proximity restriction is a huge amount of separation as well as reading in both elements "a" and "b". </p>

<p>I really doubt that pycharm has an issue processing more than 55,000 lines of output but I dont know... My guess is that I either botched the code, or it could be a windows 10 problem?</p>

<p>Thanks in advance for any help. I am pretty new to this so anything advice will surely be useful. </p>

<p>As dblclik mentions in his comment on the post, there are certainly ways to  make you code more efficient, avoiding the full computation of the nested for-loop. However I do not think that this will help you solve the error-message, that you are getting:</p>

<p>I do not think that PyCharm has an issue processing n lines of code as you are mentioning, I rather suspect, that it is the fact, that you are printing, i.e. <em>outputting</em>, all of you output to PyCharm, which especially when printing the distance of not-so-close x y's requires memory.</p>

<p>I would suggest that you instead of printing your results try to save it to a .txt file or in a distance matrix. In this way you are also able save the output of calculations!</p>

<p><strong>Example:</strong> Saving results to list</p>

<pre><code>results = []
...
for coor1 in data1:
    for coor2 in data2:
        distance = d(a, b)
        if distance &lt; thresh:
            results.append((str(coor1), str(coor2), "True", str(distance)))

        else:
            results.append((str(coor1), str(coor2), "False", str(distance)))            
</code></pre>

<p><strong>Example:</strong> Saving to textfile</p>

<pre><code>results = open("results.txt", "w")
...
for coor1 in data1:
    for coor2 in data2:
        distance = d(a, b)
        if distance &lt; thresh:

            results.write("\t".join([str(coor1), str(coor2), "True", str(distance)])+"\n")

        else:
            results.write("\t".join([str(coor1), str(coor2), "False", str(distance)])+"\n")                
results.close()
</code></pre>

<p>I hope this method will help you run your script fully through!</p>

python numpy coordinates pycharm data-analysis