Retrieving tail text from html
<p>Python 2.7 using lxml</p>

<p>I have some annoyingly formed html that looks like this:</p>

<pre><code>&lt;td&gt;
&lt;b&gt;"John"
&lt;/b&gt;
&lt;br&gt;
"123 Main st.
"
&lt;br&gt;
"New York
"
&lt;b&gt;
"Sally"
&lt;/b&gt;
&lt;br&gt;
"101 California St.
"
&lt;br&gt;
"San Francisco
"
&lt;/td&gt;
</code></pre>

<p>So basically it's a single td with a ton of stuff in it. I'm trying to compile a list or dict of the names and their addresses.</p>

<p>So far what I've done is gotten a list of nodes with names using <code>tree.xpath('//td/b')</code>. So let's assume I'm currently on the <code>b</code> node for John. </p>

<p>I'm trying to get <code>whatever.xpath('string()')</code> for everything following the current node but preceding the next <code>b</code> node (Sally). I've tried a bunch of different xpath queries but can't seem to get this right. In particular, any time I use an <code>and</code> operator in an expression that has no <code>[]</code> brackets, it returns a bool rather than a list of all nodes meeting the conditions. Can anyone help out?</p>

<p>What not use getchildren function from view of each td. For example:</p>

<pre><code>from lxml import html

s = """
&lt;td&gt;
&lt;b&gt;"John"
&lt;/b&gt;
&lt;br&gt;
"123 Main st.
"
&lt;br&gt;
"New York
"
&lt;b&gt;
"Sally"
&lt;/b&gt;
&lt;br&gt;
"101 California St.
"
&lt;br&gt;
"San Francisco
"
&lt;/td&gt;
"""

records = []
cur_record = -1
cur_field = 1

FIELD_NAME = 0
FIELD_STREET = 1
FIELD_CITY = 2

doc = html.fromstring(s)
td = doc.xpath('//td')[0]
for child in td.getchildren():
    if child.tag == 'b':
        cur_record += 1
        record = dict()
        record['name'] = child.text.strip()
        records.append(record)
        cur_field = 1
    elif child.tag == 'br':
        if cur_field == FIELD_STREET:
            records[cur_record]['street'] = child.tail.strip()
            cur_field += 1
        elif cur_field == FIELD_CITY:
            records[cur_record]['city'] = child.tail.strip()
</code></pre>

<p>And the results are:</p>

<pre><code>records = [
           {'city': '"New York\n"', 'name': '"John"\n', 'street': '"123 Main st.\n"'},
           {'city': '"San Francisco\n"', 'name': '\n"Sally"\n', 'street': '"101 California St.\n"'}
          ]
</code></pre>

<p>Note you should use <code>tag.tail</code> if you want to get text of some non-close html tag, e.g., <code>&lt;br&gt;</code>.</p>

<p>Hope this would be helpful.</p>

python xpath lxml