Why do I get garbage in the file when using "w+" instead of "a+" when the filehandles are stored in a Dict?
<p>I wrote a function that takes a list of items with several fields and write each item to one or several files depending on the content of some of the fields.</p>

<p>The name of the files is based on the content of those fields, so for example an item with value <code>AAA</code> in the field <code>rating</code> and <code>Spain</code> in the field <code>country</code> will end up in the files  <code>AAA_firms.txt</code>, <code>Spain_firms.txt</code> and <code>Spain_AAA_firms.txt</code> (just an example, not the real case).</p>

<p>When I first coded it I used <code>'w+'</code> as the mode to open the files, what I got was that most of the content of the files seemed to be corrupt, <code>^@</code> was the characters I had in the file, and only a few correct entries at the end of the file. For example we are talking of a file of more than 3500 entries with only less than 100 entries at the end being legible, the rest of the file was that <code>^@</code> characters.</p>

<p>I could not find the cause so I made it in a different way, I stored all the entries in lists in the dict and then wrote each list to a file in one pass, again opening the file with <code>w+</code>, and this worked fine but I was left with the curiosity of what happened.</p>

<p>Among other things I tried to change the <code>'w+'</code> to <code>'a+'</code>, and that works!</p>

<p>I would like to know the exact difference that makes <code>'w+'</code> work erratically and <code>'a+'</code> work fine.</p>

<p>I left the code below with the mode set to <code>'w+'</code> (this way it writes what seems to be garbage to the file).</p>

<p>The code is not 100% real, I had to modify names and is part of class (the source list itself, actually a dict wrapper as you can guess from the code here).</p>

<pre><code>def extractLists(self, outputDir, filenameprefix):
    totalEntries = 0
    aKey = "rating"
    bKey = "country"
    nameKey = "name"
    representativeChars = 2
    fileBase = outputDir + "/" + filenameprefix
    filenameAll = fileBase + "_ALL.txt"

    xLists = dict()

    for item in self.content.values():
        if (item[aKey] != aKey):
            totalEntries = totalEntries + 1
            filenameA = fileBase + "_" + item[aKey]+ "_ANY.txt"
            filenameB = fileBase + "_ANY_" + item[bKey][0:representativeBuildingChars]+ ".txt"
            filenameAB = fileBase + "_" + item[aKey]+ "_" + item[bKey][0:representativeBuildingChars] + ".txt" 
            xLists.setdefault(filenameAll,open(filenameAll,"w+")).write(item[nameKey]+"\n")
            mailLists.setdefault(filenameA,open(filenameA,"w+")).write(item[nameKey]+"\n")
            mailLists.setdefault(filenameB,open(filenameB,"w+")).write(item[nameKey]+"\n")
            mailLists.setdefault(filenameAB,open(filenameAB,"w+")).write(item[nameKey]+"\n")

    for fileHandle in mailLists.values():
        fileHandle.close()

    print(totalEntries)  
    return totalEntries
</code></pre>

<p>You are <em>reopening</em> the file objects each time in the loop, even if already present in the dictionary. The expression:</p>

<pre><code>mailLists.setdefault(filenameA,open(filenameA,"w+"))
</code></pre>

<p>opens the file <em>first</em>, as both arguments to <code>setdefault()</code> need to be available. Using <code>open(..., 'w+')</code> <em>truncates the file</em>.</p>

<p>This is fine when you do so for the first time the filename is not yet present, but all subsequent times, you just truncated a file for which there is still an open file handle. That already-existing open file handle in the dictionary has a file writing position, and continues to write from that position. Since the file just has been truncated, this leads to the behaviour you observed; corrupted file contents. You'll see multiple entries written as data could still be buffered; only data already flushed to disk is lost.</p>

<p>See this short demo (executed on OSX, different operating systems and filesystems can behave differently):</p>

<pre><code>&gt;&gt;&gt; with open('/tmp/testfile.txt', 'w') as f:
...     f.write('The quick brown fox')
...     f.flush()  # flush the buffer to disk
...     open('/tmp/testfile.txt', 'w')  # second open call, truncates
...     f.write(' jumps over the lazy fox')
...
&lt;open file '/tmp/testfile.txt', mode 'w' at 0x10079b150&gt;
&gt;&gt;&gt; with open('/tmp/testfile.txt', 'r') as f:
...     f.read()
...
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 jumps over the lazy fox'
</code></pre>

<p>Opening the files in <code>a</code> append mode doesn't truncate, which is why that change made things work.</p>

<p>Don't keep opening files, only do so when the file is <em>actually missing</em>. You'll have to use an <code>if</code> statement for that:</p>

<pre><code>if filenameA not in mailLists:
    mailLists[filenameA] = open(filenameA, 'w+')
</code></pre>

<p>I'm not sure why you are using <code>+</code> in the filemode however, since you don't appear to be reading from any of the files.</p>

<p>For <code>filenameAll</code>, that variable name never changes and you don't need to open that file in the loop at all. Move that <em>outside</em> of the loop and open just once.</p>

python io