Beautifulsoup get text based on nextSibling tag name
<p>I'm scraping multiple pages that all have a similar format, but it changes a little here and there and there are no classes to use to search for what I need.</p>

<p>The format looks like this:</p>

<pre><code>&lt;div id="mainContent"&gt;

    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;p&gt;Some Text I don't want&lt;/p&gt;
    &lt;span&gt; More text I don't want&lt;/span&gt;
    &lt;ul&gt;...unordered-list items..&lt;/ul&gt;

    &lt;p&gt;Text I WANT&lt;/p&gt;
    &lt;ol&gt;...ordered-list items..&lt;/ol&gt;

    &lt;p&gt;Text I WANT&lt;/p&gt;
    &lt;ol&gt;...ordered-list items..&lt;/ol&gt;

&lt;/div&gt;
</code></pre>

<p>The number of ordered/unordered lists and other tags changes depending on the page, but what stays the same is I always want the text from the <code>&lt;p&gt;</code> tag that is the previous sibling of the <code>&lt;ol&gt;</code> tag.</p>

<p>What I'm trying (and isn't working) is:</p>

<pre><code>main = soup.find("div", {"id":"mainContent"})

for d in main.children:
    if d.name == 'p' and d.nextSibling.name == 'ol':
        print(d.text)
    else:
        print("fail")
</code></pre>

<p>The out put of this is <code>fail</code> for every iteration. In trying to figure out why this isn't working I tried:</p>

<pre><code>for d in main.children:
    if d.name == 'p':
        print(d.nextSibling.name)
    else:
        print("fail")
</code></pre>

<p>The output of this is something like:</p>

<pre><code>fail
None
fail
None
fail
None
fail
fail
fail
fail
fail
None
fail
</code></pre>

<p>etc...</p>

<p>Why is this not working like I think it would? How could I get the text from a <code>&lt;p&gt;</code> element <em>only</em> if the next tag is <code>&lt;ol&gt;</code>?</p>

<p>You want only the <code>p</code> tags which are before <code>ol</code> tag. Find the <code>ol</code> tags first and then find the previous <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#tag" rel="nofollow">Tag</a> objects which are in this case, the <code>p</code> tag. Now your code is not working because, there is a newline between the <code>Tag</code> elements which are <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#navigablestring" rel="nofollow">NavigableString</a> type objects. And <code>d.nextSibling</code> yields you those newlines also. So You have to check the type of the object here. </p>

<pre><code>from bs4 import Tag
# create soup
# find the ols
ols = soup.find_all('ol')
for ol in ols:
     prev = ol.previous_sibling
     while(not isinstance(prev, Tag)):
         prev = prev.previous_sibling
     print(prev.text)
</code></pre>

<p>This will give you the text you want.</p>

<pre><code>Text I WANT
Text I WANT
</code></pre>

python html beautifulsoup