Error tokenizing data
<p>This is my code:</p>

<pre><code>import pandas
import datetime
from decimal import Decimal

file_ = open('myfile.csv', 'r')
result = pandas.read_csv(
    file_, header=None,
    names=('sec', 'date', 'sale', 'buy'),
    usecols=('date', 'sale', 'buy'),
    parse_dates=['date'],
    iterator=True,
    chunksize=100,
    compression=None,
    engine="c",
    date_parser=lambda dt: datetime.datetime.strptime(dt, '%Y%m%d %H:%M:%S.%f'),
    converters={'sale': (lambda u: Decimal(u)), 'buy': (lambda u: Decimal(u))}
)
</code></pre>

<p>And then I try...</p>

<pre><code>result.get_chunk()
</code></pre>

<p>Only to get an error like this:</p>

<pre><code>CParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4
</code></pre>

<p>From a file like this (I just show the first 4 lines - the file has no header, and all the lines have this format):</p>

<pre><code>EUR/USD,20160701 00:00:00.071,1.11031,1.11033
EUR/USD,20160701 00:00:00.255,1.11031,1.11033
EUR/USD,20160701 00:00:00.256,1.11025,1.11033
EUR/USD,20160701 00:00:00.258,1.11027,1.11033
... &gt; l0.000.000 lines like these
</code></pre>

<p>My intention is to get an object to iterate by chunks and not have the whole crap in memory (the actual file has 560mb!). I want to discard the first column (there are 4 columns but since this file has the same value in the first column, I want to discard such column). I want to keep columns 1, 2, and 3 (discarding 0) as date, sale, and purchase price.</p>

<p>Actually this is my first attempt with pandas, since the former solution used standard Python csv module, and takes a lot of time.</p>

<p>What am I missing? Why am I getting such error?</p>

<pre><code>#try this code
import pandas as pd
import numpy as np
import csv

# To print only three columns , create a data frame,to do that give names to columns in csv file with ',' as seperator
myfile.csv:
sec,date,sale,buy
EUR/USD,20160701 00:00:00.071,1.11031,1.11033
EUR/USD,20160701 00:00:00.255,1.11031,1.11033
EUR/USD,20160701 00:00:00.256,1.11025,1.11033
EUR/USD,20160701 00:00:00.258,1.11027,1.11033

data = pd.read_csv('myfile.csv',sep=',')
df = pd.DataFrame({'date':data.date,'sale':data.sale,'buy':data.buy})
print(df)

output:
       buy                   date     sale
0  1.11033  20160701 00:00:00.071  1.11031
1  1.11033  20160701 00:00:00.255  1.11031
2  1.11033  20160701 00:00:00.256  1.11025
3  1.11033  20160701 00:00:00.258  1.11027
</code></pre>

python pandas