Reading through a bunch of .gz files error: Error -3 while decompressing: invalid code lengths set
<p>I am trying to parse a bunch of .gz json files (the files are text files) by using the same function using multiprocessing in Python 2.7.10. However, almost at the very end of parsing each line in these files, it produces this error:</p>

<p><code>error: Error -3 while decompressing: invalid code lengths set</code></p>

<p>and stops the execution.</p>

<p>This is my code:</p>

<pre><code>import gzip
import json
from multiprocessing import Pool, cpu_count

def build_list(file_name):

    count = 0

    try:
        json_file = gzip.open(file_name, "r")
    except Exception as e:
        print e
    else:

        # Data parsing
        for line in json_file:
            try:
                row = json.loads(line)
            except Exception as e:
                print e
            else:                
                count += 1

if __name__ == "__main__":

    files = ["h1.json.gz", "h2.json.gz", "h3.json.gz", "h4.json.gz", "h5.json.gz"]

    pool = Pool(processes=cpu_count()-1)
    pool.map(build_list, files)
</code></pre>

<p>It is important to clarify that the program starts running well and that the files are assigned at each processor when I check with <code>top</code>. I also check the integrity of the files with <code>gunzip -t</code> and they seems to be well formed. Also I did not see any exception raised before the error. Do you have any ideas on how can I fix it? Thanks in advance. </p>

<p>Read in binary mode:</p>

<pre><code>gzip.open(file_name, "rb")
</code></pre>

<p>Reading in text mode may mangle the data (as it is not text) on some platforms, and will cause odd errors such as this.</p>

python json python-2.7 multiprocessing gzip