Obtain csv-like parse AND line length byte count?
<p>I'm familiar with the <code>csv</code> Python module, and believe it's necessary in my case, as I have some fields that contain the delimiter (<code>|</code> rather than <code>,</code>, but that's irrelevant) within quotes.</p>

<p>However, I am also looking for the byte-count length of each original row, <em>prior</em> to splitting into columns. I can't count on the data to always quote a column, and I don't know if/when <code>csv</code> will strip off outer quotes, so I don't think (but might be wrong) that simply joining on my delimiter will reproduce the original line string (less CRLF characters). Meaning, I'm not positive the following works:</p>

<pre><code>with open(fname) as fh:
    reader = csv.reader(fh, delimiter="|")
    for row in reader:
        original = "|".join(row) ## maybe?
</code></pre>

<p>I've tried looking at <code>csv</code> to see if there was anything in there that I could use/monkey-patch for this purpose, but since <code>_csv.reader</code> is a <code>.so</code>, I don't know how to mess around with that.</p>

<p>In case I'm dealing with an XY problem, my ultimate goal is to read through a CSV file, extracting certain fields and their overall file offsets to create a sort of look-up index. That way, later, when I have a list of candidate values, I can check each one's file-offset and <code>seek()</code> there, instead of chugging through the whole file again. As an idea of scale, I might have 100k values to look up across a 10GB file, so re-reading the file 100k times doesn't feel efficient to me. I'm open to other suggestions than the CSV module, but will still need <code>csv</code>-like intelligent parsing behavior. </p>

<p>EDIT: Not sure how to make it more clear than the title and body already explains - simply <code>seek()</code>-ing on a file handle isn't sufficient because I <strong>also</strong> need to parse the lines as a <code>csv</code> in order to pull out additional information.</p>

<p>You can't subclass <code>_csv.reader</code>, but the <em><code>csvfile</code></em> argument to the <code>csv.reader()</code> <a href="https://docs.python.org/3/library/csv.html#csv.reader" rel="nofollow">constructor</a> only has to be a "file-like object". This means you could supply an instance of your own class that does some preprocessingâsuch as remembering the length of the last line read and file offset. Here's an implementation showing exactly that. Note that the line length does <em>not</em> include the end-of-line character(s). It also shows how the offsets to each line/row could be stored and used after the file is read.</p>

<pre><code>import csv

class CSVInputFile(object):
    """ File-like object. """
    def __init__(self, file):
        self.file = file
        self.offset = None
        self.linelen = None
    def __iter__(self):
        return self
    def __next__(self):
        offset = self.file.tell()
        data = self.file.readline()
        if not data:
            raise StopIteration
        self.offset = offset
        self.linelen = len(data)
        return data
    next = __next__

offsets = []  # remember where each row starts
fname = 'unparsed.csv'
with open(fname) as fh:
    csvfile = CSVInputFile(fh)
    for row in csv.reader(csvfile, delimiter="|"):
        print('offset: {}, linelen: {}, row: {}'.format(
            csvfile.offset, csvfile.linelen, row))  # file offset and length of row
        offsets.append(csvfile.offset)  # remember where each row started
</code></pre>

python csv