How to get lineno of "end-of-statement" in Python ast
<p>I am trying to work on a script that manipulates another script in Python, the script to be modified has structure like:</p>

<pre><code>class SomethingRecord(Record):
    description = 'This records something'
    author = 'john smith'
</code></pre>

<p>I use <code>ast</code> to locate the <code>description</code> line number, and I use some code to change the original file with new description string base on the line number. So far so good.</p>

<p>Now the only issue is <code>description</code> occasionally is a multi-line string, e.g.</p>

<pre><code>    description = ('line 1'
                   'line 2'
                   'line 3')
</code></pre>

<p>or</p>

<pre><code>    description = 'line 1' \
        'line 2' \
        'line 3'
</code></pre>

<p>and I only have the line number of the first line, not the following lines. So my one-line replacer would do</p>

<pre><code>    description = 'new value'
        'line 2' \
        'line 3'
</code></pre>

<p>and the code is broken. I figured that if I know both the lineno of start and end/number of lines of <code>description</code> assignment I could repair my code to handle such situation. How do I get such information with Python standard library?</p>

<p>Indeed, the information you need is not stored in the <code>ast</code>. I don't know the details of what you need, but it looks like you could use the <code>tokenize</code> module from the standard library. The idea is that every logical Python statement is ended by a <code>NEWLINE</code> token (also it could be a semicolon, but as I understand it is not your case). I tested this approach with such file:</p>

<pre><code># first comment
class SomethingRecord:
    description = ('line 1'
                   'line 2'
                   'line 3')

class SomethingRecord2:
    description = ('line 1',
                   'line 2',
                   # comment in the middle

                   'line 3')

class SomethingRecord3:
    description = 'line 1' \
                  'line 2' \
                  'line 3'
    whatever = 'line'

class SomethingRecord3:
    description = 'line 1', \
                  'line 2', \
                  'line 3'
                  # last comment
</code></pre>

<p>And here is what I propose to do:</p>

<pre><code>import tokenize
from io import BytesIO
from collections import defaultdict

with tokenize.open('testmod.py') as f:
    code = f.read()
    enc = f.encoding

rl = BytesIO(code.encode(enc)).readline
tokens = list(tokenize.tokenize(rl))

token_table = defaultdict(list)  # mapping line numbers to token numbers
for i, tok in enumerate(tokens):
    token_table[tok.start[0]].append(i)

def find_end(start):
    i = token_table[start][-1]  # last token number on the start line
    while tokens[i].exact_type != tokenize.NEWLINE:
        i += 1
    return tokens[i].start[0]

print(find_end(3))
print(find_end(8))
print(find_end(15))
print(find_end(21))
</code></pre>

<p>This prints out:</p>

<pre><code>5
12
17
23
</code></pre>

<p>This seems to be correct, you could tune this approach depending on what exactly you need. <code>tokenize</code> is more verbose than <code>ast</code> but also more flexible. Of course the best approach is to use them both for different parts of your task.</p>

<hr>

<p><strong>EDIT:</strong> I tried this in Python 3.4, but I think it should also work in other versions.</p>

python abstract-syntax-tree