Extracting text from find_next_sibling(), BeautifulSoup
<p>I am trying to extract the description of the Chinese character from this website: <a href="http://www.hsk.academy/en/hsk_1" rel="nofollow">http://www.hsk.academy/en/hsk_1</a></p>

<p>Example html:
</p>

<pre><code>        &lt;tr&gt;
            &lt;td&gt;
                &lt;span class="hanzi"&gt;&lt;a href="/en/characters/%E7%88%B1"&gt;ç±&lt;/a&gt;&lt;/span&gt;
                &lt;br/&gt;Ã i&lt;/td&gt;
            &lt;td&gt;to love; affection; to be fond of; to like&lt;/td&gt;
        &lt;/tr&gt;
</code></pre>

<p>I would like the last td tag's text be put into a list for each description of the character. However, currently I am given the whole tag including the tags themselves. I can't .text the find_next_sibling(): AttributeError: 'NoneType' object has no attribute 'text'.</p>

<p>This is my code:</p>

<pre><code>for item in soup.find_all("td"):   
        EnglishItem = item.find_next_sibling()
        if EnglishItem:           
            if not any(EnglishItem in s for s in EnglishDescriptionList):
                EnglishDescriptionList.insert(count, EnglishItem)
                count += 1
            print EnglishDescriptionList
</code></pre>

<p>Try this:</p>

<pre><code>english_descriptions = []
table = soup.find('table', id='flat_list')
for e in table.select('.hanzi'):
    english_desc = e.parent.find_next_sibling().text
    if not any(english_desc in s for s in english_descriptions):
        english_descriptions.append(english_desc)
</code></pre>

<p>This selects (finds) all tags of class <code>hanzi</code> (within the table with <code>id="flat_list"</code>) which will be the <code>&lt;span&gt;</code> tags. Then the parent of each <code>&lt;span&gt;</code> is accessed - this is the first <code>&lt;td&gt;</code> in each row. Finally the next sibling is accessed and this is the target tag that contains the English description.</p>

<p>You can do away with the <code>count</code> and just append items to the list with</p>

<pre><code>english_descriptions.append()
</code></pre>

<p>Also, I don't think that you need to check whether the current english description is a substring of an existing one (is that what you're trying to do?). If not you can simplify to this list comprehension:</p>

<pre><code>table = soup.find('table', id='flat_list')
english_descriptions = [e.parent.find_next_sibling().text for e in table.select('.hanzi')]
</code></pre>

python python-2.7 beautifulsoup