Pandas missing rows when importing using delimiter ="|"
<p>I have a dataset with 51,347 rows. When import the data using pandas and set the delimiter to "|" , I lose 394 rows. </p>

<pre><code> import pandas as pd
 df = pd.read_csv("Basin11.txt", sep='|', error_bad_lines=False, 
 dtype={'Start Date': str, 'Greater Than/Less Than': str, 
'Parameter Code': float, 'Start Time': str, 'Start Depth': float, 'Composite Category': str,
'Composite Type': str})
 print(len(df.index))
</code></pre>

<p>If I remove the sep variable, the data won't load as multiple columns but will load the proper number of rows. It only seems to be an issue for this file. 
<a href="http://newcoastalatlas.tamug.edu/download/Basin/basin11.txt" rel="nofollow">Basin11.txt File</a></p>

<p>Does anyone know why I'm losing data?</p>

<p>I started going through your input file and found a number of errors that may be leading to the "missing rows".</p>

<p>The comments line 3491 and 9805 have an opening <code>"</code> but is missing the closing <code>"</code>.  This would cause matching issues, including the following rows as part of the comment body.  As I started fixing those, the line counts started going up.  There are probably more cases of this.</p>

<p>Also, some lines have double-double quotes (<code>""</code>) for opening and closing comments.  For example:</p>

<blockquote>
  <p>""green, med tide, 10-15 mph winds""</p>
</blockquote>

<p>Edit:  I added the following code:</p>

<pre><code>for comment in df['Comments'].values:
    print(comment)
</code></pre>

<p>Then ran <code>python3 sample.py | grep '|' | wc -l</code>, to find the number of comments that contained <code>|</code>, and got 394 (The number of rows that you are missing)</p>

python pandas