How to manipulate a huge CSV file in python
<p>I have a CSV file more than 16G, each row is the text data. When I was encoding (e.g. one-hot-encode) the whole CSV file data, my process was killed due to the memory limitation. Is there a way to process this kind of "big data"? </p>

<p>I am thinking that split the whole CSV file into multiple "smaller" files, then append them to another CSV file, is that a correct way to handle the huge CSV file?</p>

<p>This has been discussed in  <a href="http://stackoverflow.com/questions/33689456/reading-huge-csv-files-efficiently?rq=1">Reading huge csv files efficiently?</a></p>

<p>Probably the most reasonable thing to do with a 16GB csv file would not to load it all into memory, but read and process it line by line:</p>

<pre><code>with open(filename, "r") as f:
    lines = csv.reader(f)
    for line in lines:
        #Process the line
</code></pre>

python csv encoding bigdata