How to delete column from .csv file without reading the whole file
<p>I generate very big .csv file but now it doesn't fit the RAM. So i decided to delete some inefficient columns to reduce the file size. How can I do that? </p>

<p>I tried data = <code>pd.read_csv("file.csv", index_col=0, usecols=["id", "wall"])</code> but it still doesn't fit the RAM.</p>

<p>File is about 1.5GB, RAM is 8GB. </p>

<p>Instead of deleting columns, you can also read specific columns from csv file using a <code>DictReader</code> (if you're not using <code>Pandas</code> ).</p>

<pre><code>import csv
from StringIO import StringIO

columns = 'AAA,DDD,FFF,GGG'.split(',')


testdata ='''\
AAA,bbb,ccc,DDD,eee,FFF,GGG,hhh
1,2,3,4,50,3,20,4
2,1,3,5,24,2,23,5
4,1,3,6,34,1,22,5
2,1,3,5,24,2,23,5
2,1,3,5,24,2,23,5
'''

reader = csv.DictReader(StringIO(testdata))

desired_cols = (tuple(row[col] for col in columns) for row in reader)
</code></pre>

<p>Output:</p>

<pre><code>&gt;&gt;&gt; list(desired_cols)
[('1', '4', '3', '20'),
 ('2', '5', '2', '23'),
 ('4', '6', '1', '22'),
 ('2', '5', '2', '23'),
 ('2', '5', '2', '23')]
</code></pre>

<p>Source: <a href="http://stackoverflow.com/a/20065131/6633975">http://stackoverflow.com/a/20065131/6633975</a></p>

<p><strong>Using Pandas:</strong></p>

<p>Here is an example illustrating the answer given by EdChum. There is a lot of additional options to load a CSV file, check the <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html" rel="nofollow">API reference</a>.</p>

<pre><code>import pandas as pd


raw_data = {'first_name': ['Steve', 'Guido', 'John'],
        'last_name': ['Jobs', 'Van Rossum', "von Neumann"]}
df = pd.DataFrame(raw_data)
# Saving data without header
df.to_csv(path_or_buf='test.csv', header=False)
# Telling that there is no header and loading only the first name
df = pd.read_csv(filepath_or_buffer='test.csv', header=None, usecols=[1], names=['first_name'])
df

  first_name
0      Steve
1      Guido
2       John
</code></pre>

python csv pandas