Iterating over large line separated file without freezing
<p>I'm trying to implement a Strongly Connected Graph search algorithm and am trying to load a large file with edges of the graph into memory to do it.</p>

<p>The file comes as:</p>

<blockquote>
  <p> 1 2 // i.e. a directed edge from 1 to 2 </p>
  
  <p> 1 3 </p>
  
  <p> 2 3 </p>
  
  <p> 3 1 </p>
  
  <p>...</p>
</blockquote>

<p>For this part of the algorithm I need to reverse the graph, and I am hoping to store each node in a dictionary along with some other values which will be important for the algorithm.</p>

<p>This code works for smaller files but just stalls/hangs for my large file, which is 72.7mb, I would appreciate any suggestions to make it work on large files:</p>

<pre><code>def begin():
    graph = {}
    for line in open('/home/edd91/Documents/SCC.txt'):
        tempArray = []
        for e in line.split():
            if e != ' ' and e!='\n':
                tempArray.append(int(e))            
        if tempArray[1] in graph.keys():
            graph[tempArray[1]]['g'].append(tempArray[0])
        else:
            graph[tempArray[1]] = {'g': [tempArray[0],], 's': False, 't': None, 'u': None }
    print len(graph)
</code></pre>

<p>Not much to be done, but since it's a little too complex, speed suffers.
You can improve the program speed like this:</p>

<pre><code>def begin():
    graph = {}
    for line in open('/home/edd91/Documents/SCC.txt'):
        # consider only first 2 fields: avoids the loop and the append: that's the best saver
        # there must not be 2 spaces between the numbers (in your original program neither so not a problem)
        tempArray = line.split()  
        v1 = tempArray[1]  # copy to get rid of array
        if v1 in graph:  # no need to use .keys()
            graph[v1]['g'].append(tempArray[0])
        else:
            graph[v1] = {'g': [tempArray[0],], 's': False, 't': None, 'u': None }
    print len(graph)
</code></pre>

<p>You can compare speed vs your original program with a medium-sized file first.</p>

python graph