csv write doesn't work correctly
<p>Any idea why is this always writing the same line in output csv?</p>

<pre><code> 21 files = glob.glob(path)
 22 csv_file_complete = open("graph_complete_reddit.csv", "wb")
 23 stat_csv_file = open("test_stat.csv", "r")
 24 csv_reader = csv.reader(stat_csv_file)
 25 lemmatizer = WordNetLemmatizer()
 26 for file1, file2 in itertools.combinations(files, 2):
 27         with open(file1) as f1:
 28                 print(file1)
 29                 f1_text = f1.read()
 30                 f1_words = re.sub("[^a-zA-Z]", ' ', f1_text).lower().split()
 31                 f1_words = [str(lemmatizer.lemmatize(w, wordnet.VERB)) for w in f1_words if w not in stopwords]
 32                 print(f1_words)
 33         f1.close()
 34         with open(file2) as f2:
 35                 print(file2)
 36                 f2_text = f2.read()
 37                 f2_words = re.sub("[^a-zA-Z]", ' ', f2_text).lower().split()
 38                 f2_words = [str(lemmatizer.lemmatize(w, wordnet.VERB)) for w in f2_words if w not in stopwords]
 39                 print(f2_words)
 40         f2.close()
 41 
 42         a_complete = csv.writer(csv_file_complete, delimiter=',')
 43         print("*****")
 44         print(file1)
 45         print(file2)
 46         print("************************************")
 47 
 48         f1_head, f1_tail = os.path.split(file1)
 49         print("************")
 50         print(f1_tail)
 51         print("**************")
 52         f2_head, f2_tail = os.path.split(file2)
 53         print(f2_tail)
 54         print("********************************")
 55         for row in csv_reader:
 56             if f1_tail in row:
 57                 file1_file_number = row[0]
 58                 file1_category_number = row[2]
 59             if f2_tail in row:
 60                 file2_file_number = row[0]
 61                 file2_category_number = row[2]
 62 
 63         row_complete = [file1_file_number, file2_file_number, file1_category_number, file2_category_number ]
 64         a_complete.writerow(row_complete)
 65 
 66 csv_file_complete.close()
</code></pre>

<p>Those prints show different filenames!</p>

<p>This is test_stat.csv file which the code uses as input:</p>

<pre><code>  1 1,1bmmoc.txt,1
  2 2,2b3u1a.txt,1
  3 3,2mf64u.txt,2
  4 4,4x74k3.txt,5
  5 5,lsspe.txt,3
  6 6,qbimg.txt,4
  7 7,w95fm.txt,2
</code></pre>

<p>And here's what the code outputs:</p>

<pre><code>  1 7,4,2,5
  2 7,4,2,5
  3 7,4,2,5
  4 7,4,2,5
  5 7,4,2,5
  6 7,4,2,5
  7 7,4,2,5
  8 7,4,2,5
  9 7,4,2,5
 10 7,4,2,5
 11 7,4,2,5
 12 7,4,2,5
 13 7,4,2,5
 14 7,4,2,5
 15 7,4,2,5
 16 7,4,2,5
 17 7,4,2,5
 18 7,4,2,5
 19 7,4,2,5
 20 7,4,2,5
 21 7,4,2,5
</code></pre>

<p>please comment or suggest fixes.</p>

<p>You're never rewinding <code>stat_csv_file</code>, so eventually, your loop over <code>csv_reader</code> (which is a wrapper around <code>stat_csv_file</code>) isn't looping at all, and you write whatever you found on the last loop. Basically, the logic is:</p>

<ol>
<li>On first loop, look through all of <code>csv_reader</code>, find hit (though you keep looking even when you find it, exhausting the file), write hit</li>
<li>On all subsequent loops, the file is exhausted, so the inner search loop doesn't even execute, and you end up writing the same values as last time</li>
</ol>

<p>The slow, but direct way to fix this is to add <code>stat_csv_file.seek(0)</code> before you search it:</p>

<pre><code> 53         print(f2_tail)
 54         print("********************************")
            stat_csv_file.seek(0)  # Rewind to rescan input from beginning
 55         for row in csv_reader:
 56             if f1_tail in row:
 57                 file1_file_number = row[0]
 58                 file1_category_number = row[2]
 59             if f2_tail in row:
 60                 file2_file_number = row[0]
 61                 file2_category_number = row[2]
</code></pre>

<p>A likely better approach would be to load the input CSV into a <code>dict</code> once, then perform lookup there as needed, avoiding repeated (slow) I/O in favor of fast <code>dict</code> lookup. The cost would be higher memory use; if the input CSV is small enough, that's not an issue, if it's huge, you may need to use a proper database to get the rapid lookup without blowing memory.</p>

<p>It's a little unclear what the logic should be here, since your inputs and outputs don't align (your output should start with a repeated digit, but it doesn't for some reason?). But if the intent is that the input contains <code>file_number, file_tail, category_number</code>, then you could begin your code (above the top level loop) with:</p>

<pre><code># Create mapping from second field to associated first and third fields
tail_to_numbers = {ftail: (fnum, cnum) for fnum, ftail, cnum in csv_reader}
</code></pre>

<p>Then replace:</p>

<pre><code>    for row in csv_reader:
        if f1_tail in row:
            file1_file_number = row[0]
            file1_category_number = row[2]
        if f2_tail in row:
            file2_file_number = row[0]
            file2_category_number = row[2]

    row_complete = [file1_file_number, file2_file_number, file1_category_number, file2_category_number ]
    a_complete.writerow(row_complete)
</code></pre>

<p>with the simpler and much faster:</p>

<pre><code>try:
    file1_file_number, file1_category_number = tail_to_numbers[f1_tail]
    file2_file_number, file2_category_number = tail_to_numbers[f2_tail]
except KeyError:
    # One of the tails wasn't found in the lookup dict, so don't output
    # (variables would be stale or unset); optionally emit some error to stderr
    continue
else:
    # Found both tails, output associated values
    row_complete = [file1_file_number, file2_file_number, file1_category_number, file2_category_number]
    a_complete.writerow(row_complete)
</code></pre>

python file csv itertools