Read data from large tar.gz file from the website
<p>1) How should I read the data from all the csv files in the tar.gz file on website and write them to the CSVs on a folder in the most memory and space efficient way? 
2) How can I loop it to go over all the CSVs in the tar.gz file?
3) Since the CSV files are huge, how can I loop it to read and write, let's say, 1 million rows at a time?</p>

<p>I have gone only so far using the codes on other stackoverflow answers!</p>

<pre><code>import pandas as pd
import urllib2
import tarfile
url='https://ghtstorage.blob.core.windows.net/downloads/mysql-2016-08-01.tar.gz'
r=urllib2.Request(url)
o=urllib2.urlopen(r)

thetarfile=tarfile.open(o, mode='r:gz')
thetarfile.close()
</code></pre>

<ol>
<li>Download an archive to your local storage.</li>
<li>Display the list of files in the archive. Run <strong>man tar</strong> to see options for command line.</li>
<li>Extract files one by one from the archive.</li>
<li>Use SAX xml parser <a href="https://docs.python.org/2/library/xml.sax.reader.html" rel="nofollow">https://docs.python.org/2/library/xml.sax.reader.html</a>.</li>
<li>Remove file after parsing.</li>
<li>Remove the archive.</li>
</ol>

python python-2.7 csv urllib2 tar