How to manipulate a huge CSV file in python
<p>I have a CSV file more than 16G, each row is the text data. When I was encoding (e.g. one-hot-encode) the whole CSV file data, my process was killed due to the memory limitation. Is there a way to process this kind of "big data"? </p>

<p>I am thinking that split the whole CSV file into multiple "smaller" files, then append them to another CSV file, is that a correct way to handle the huge CSV file?</p>

<p>Your question does not state what language you are using to handle this CSV file. I will reply using C#, but I imagine that the strategy will work equally well for Java too.</p>

<p>You can try using the <code>StreamReader</code> class to read the file line-by-line. That should take care of the read side of things.</p>

<p>Something like:</p>

<pre><code>using (var reader = new StreamReader(...))
{
    var line = string.Empty;

    while ((line != reader.ReadLine()) != null)
    {
        Process(line);
    }
}
</code></pre>

<p>NB: That's a code snippet in C#, and is more pseudo-code than actual code.</p>

<p>You should create a database using some kind of local DB technology, either SQLite or SQL Server LocalDB or even MySQL and load the data into a table or tables in that.</p>

<p>You can then write any other further processing based on the data held in the database rather than in a simple text file.</p>

python csv encoding bigdata