What to do if a query result is too big
<p>I'm using mysql connector of python3 to query from a mysql database. I have so many data that if I collect the recent 7 days' data, the size of the query will be over 10GB and therefore forced my python script to be killed. I think usually we can stream the result however I don't find a way to stream the query result in mysql conncetor. Is there any other way that I can solve the problem?</p>

<p>Don't fetch the result set in one go. You can do one of the following or the combination of these:</p>

<ol>
<li>Use <code>LIMIT, OFFSET</code> to generate multiple files using <code>SELECT into</code></li>
<li>Use Date/Hour Functions to generate multiple files using <code>SELECT into</code></li>
</ol>

<p><code>cat</code> all the generated files into one file using <code>cat hugedata_* &gt; hugedata.csv</code></p>

python mysql mysql-connector