Number of rows in 40GB tar.gz file without uncompressing it?
<p>I have over 40 gb tar.gz file at <a href="https://ghtstorage.blob.core.windows.net/downloads/mysql-2016-06-16.tar.gz" rel="nofollow">https://ghtstorage.blob.core.windows.net/downloads/mysql-2016-06-16.tar.gz</a>
How can I find the number of rows in the CSV file that is compressed inside this tar.gz file without uncompressing the entire file which might be in 100+ GBs? </p>

<blockquote>
  <p>without uncompressing the entire file which might be in 100+ GBs</p>
</blockquote>

<p>I suppose you mean without extracting the file to disk first. Here's a Python approach to achieve this:</p>

<pre><code>import tarfile as tf
import gzip as gz
from StringIO import StringIO
infile = '/path/to/mysql-2016-06-16.tar.gz'
def linecount(infile, member):
    lc = 0
    with gz.GzipFile(infile) as zipf:
        with tf.TarFile(fileobj=zipf) as tarf:
            dataf = tarf.extractfile(member)
            while dataf.readline():
               lc += 1 
            dataf.close()       
    return lc
print linecount(infile, 'test.csv')
</code></pre>

<blockquote>
  <p>it say's "filename 'test.csv' not found".</p>
</blockquote>

<p>To know what members the tar file has:</p>

<pre><code>def listmembers(infile):
    with gz.GzipFile(infile) as zipf:
        with tf.TarFile(fileobj=zipf) as tarf:
            return list(m.name for m in tarf)  
</code></pre>

<p>To count the lines of all files in the tarfile:</p>

<pre><code>for member in listmembers(infile):
    print member, linecount(infile, member)
</code></pre>

<p>Before you start it will be useful to <a href="https://en.wikipedia.org/wiki/Tar_(computing)#File_format" rel="nofollow">know how tar files are structured</a>.</p>

python csv gzip tar