Beautifulsoup get text based on nextSibling tag name
<p>I'm scraping multiple pages that all have a similar format, but it changes a little here and there and there are no classes to use to search for what I need.</p>

<p>The format looks like this:</p>

<pre><code>&lt;div id="mainContent"&gt;

    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;span&gt; More text I don't want&lt;/span&gt;
    &lt;ul&gt;...unordered-list items..&lt;/ul&gt;

    &lt;p&gt;Text I WANT&lt;/p&gt;
    &lt;ol&gt;...ordered-list items..&lt;/ol&gt;

    &lt;p&gt;Text I WANT&lt;/p&gt;
    &lt;ol&gt;...ordered-list items..&lt;/ol&gt;

&lt;/div&gt;
</code></pre>

<p>The number of ordered/unordered lists and other tags changes depending on the page, but what stays the same is I always want the text from the <code>&lt;p&gt;</code> tag that is the previous sibling of the <code>&lt;ol&gt;</code> tag.</p>

<p>What I'm trying (and isn't working) is:</p>

<pre><code>main = soup.find("div", {"id":"mainContent"})

for d in main.children:
    if d.name == 'p' and d.nextSibling.name == 'ol':
        print(d.text)
    else:
        print("fail")
</code></pre>

<p>The out put of this is <code>fail</code> for every iteration. In trying to figure out why this isn't working I tried:</p>

<pre><code>for d in main.children:
    if d.name == 'p':
        print(d.nextSibling.name)
    else:
        print("fail")
</code></pre>

<p>The output of this is something like:</p>

<pre><code>fail
None
fail
None
fail
None
fail
fail
fail
fail
fail
None
fail
</code></pre>

<p>etc...</p>

<p>Why is this not working like I think it would? How could I get the text from a <code>&lt;p&gt;</code> element <em>only</em> if the next tag is <code>&lt;ol&gt;</code>?</p>

<p>You can use a <em>css selector</em>, i.e <code>ul ~ p</code> to find all the p tags preceded by the <em>ul</em>:</p>

<pre><code>html = """&lt;div id="mainContent"&gt;

    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;span&gt; More text I don't want&lt;/span&gt;
    &lt;ul&gt;...unordered-list items..&lt;/ul&gt;

    &lt;p&gt;Text I WANT&lt;/p&gt;
    &lt;ol&gt;...ordered-list items..&lt;/ol&gt;

    &lt;p&gt;Text I WANT&lt;/p&gt;
    &lt;ol&gt;...ordered-list items..&lt;/ol&gt;

&lt;/div&gt;"""

from bs4 import BeautifulSoup

soup = BeautifulSoup(html)


print([p.text for p in soup.select("#mainContent  ul ~ p")])
</code></pre>

<p>Which will give you:</p>

<pre><code>['Text I WANT', 'Text I WANT']
</code></pre>

<p>Or find the ol's and then look for the <em>previous sibling</em> p:</p>

<pre><code>print([ol.find_previous_sibling("p").text for ol in soup.select("#mainContent ol")])
</code></pre>

<p>Which would also give you:</p>

<pre><code>['Text I WANT', 'Text I WANT']
</code></pre>

python html beautifulsoup