Managing zipping large number of files in Python using 7zip and subprocess
<p>I have a large amount of files (~100k) moved from a server to a designated folder on a daily basis. I'm running a Python script against these files. At the end of the process I want to create a number of .zip files, each of approximately 2GB.</p>

<p>The script will get a list of all files in the folder, get the size one a time using <code>os.path.getsize</code> , keep track of the total accrued file size until it gets above 2GB, it will take the list of those files and zip them in one zip archive, then reset the size accumulator and move on to the next chunk of files.</p>

<p>If there wasn't a 2GB requirement I'd just run the following and it would be super speedy.</p>

<pre><code>txt = '"C:\\Program Files\\7-Zip\\7z.exe" a -tzip %s *.txt -pSECRET'%(zipFileName)
out = subprocess.check_output(txt, shell = True)
</code></pre>

<p>But given the 2GB requirement I've tried two routes. One is to add one file at a time to the archive, but that seems to be orders of magnitude slower than just using a wildcard (I'm hypothesizing that is due to the constant re-opening of the archive).</p>

<pre><code>for file in FilesListBelow2GB:
    txt = '"C:\\Program Files\\7-Zip\\7z.exe" a -tzip %s %s -pSECRET'%(zipFileName,file)
    out = subprocess.check_output(txt, shell = True)
</code></pre>

<p>I also tried to supply a space delimited list of files to process but it errors <code>WindowsError: [Error 206] The filename or extension is too long</code>.</p>

<pre><code>txt = '"C:\\Program Files\\7-Zip\\7z.exe" a -tzip %s %s -pSECRET'%(zipFileName,' '.join(FilesListBelow2GB))
out = subprocess.check_output(txt, shell = True)
</code></pre>

<p>How can I manage this situation so that I can achive better zipping performance given the constraints I'm given (all files are in the same folder, can't delete files, zip files need to be approximately ~2GB in size)?</p>

<p>I ended up finding a workaround, where essentially I write all file names from FilesListBelow2GB to a .txt file, and then ask 7zip to use that to know which files to zip.</p>

<pre><code>outf = open('list.txt','w') 
for fileToZip in FilesListBelow2GB :
    outf.write(fileToZip+'\n')
    outf.close()

txt = '"C:\\Program Files\\7-Zip\\7z.exe" a -tzip compress.zip -i@list.txt'
out = subprocess.check_output(txt, shell = True)
</code></pre>

python python-2.7 zip subprocess