Obtain csv-like parse AND line length byte count?
<p>I'm familiar with the <code>csv</code> Python module, and believe it's necessary in my case, as I have some fields that contain the delimiter (<code>|</code> rather than <code>,</code>, but that's irrelevant) within quotes.</p>

<p>However, I am also looking for the byte-count length of each original row, <em>prior</em> to splitting into columns. I can't count on the data to always quote a column, and I don't know if/when <code>csv</code> will strip off outer quotes, so I don't think (but might be wrong) that simply joining on my delimiter will reproduce the original line string (less CRLF characters). Meaning, I'm not positive the following works:</p>

<pre><code>with open(fname) as fh:
    reader = csv.reader(fh, delimiter="|")
    for row in reader:
        original = "|".join(row) ## maybe?
</code></pre>

<p>I've tried looking at <code>csv</code> to see if there was anything in there that I could use/monkey-patch for this purpose, but since <code>_csv.reader</code> is a <code>.so</code>, I don't know how to mess around with that.</p>

<p>In case I'm dealing with an XY problem, my ultimate goal is to read through a CSV file, extracting certain fields and their overall file offsets to create a sort of look-up index. That way, later, when I have a list of candidate values, I can check each one's file-offset and <code>seek()</code> there, instead of chugging through the whole file again. As an idea of scale, I might have 100k values to look up across a 10GB file, so re-reading the file 100k times doesn't feel efficient to me. I'm open to other suggestions than the CSV module, but will still need <code>csv</code>-like intelligent parsing behavior. </p>

<p>EDIT: Not sure how to make it more clear than the title and body already explains - simply <code>seek()</code>-ing on a file handle isn't sufficient because I <strong>also</strong> need to parse the lines as a <code>csv</code> in order to pull out additional information.</p>

<p>Depending on performance requirements and the size of the data, the low tech solution is to simply read the file twice. Make a first pass where you get the length of each line, and then then you can run the data through the csv parser. On my somewhat outdated Mac I can read and count the length of 2-3 million lines in a second, which isn't a huge performance hit.</p>

python csv