beautifulsoup does not find all p with a certain class
<p>I am using beautifulsoup to find all p in a certain html page I saved locally.
My code is</p>

<pre><code>with open ("./" + str(filename) + ".txt", "r") as myfile:
    data=myfile.read().replace('\n', '')
soup = BeautifulSoup(data)
t11 = soup.findAll("p", {"class": "commentsParagraph"})
</code></pre>

<p>this codes works for part of the page, but some part of the page is loaded with ajax (which I preloaded before I saved the source), and the code is not working on it.</p>

<p>to test this I added to one of the <code>p</code> tags in the ajax portion the class <code>commentsParagraph2</code>, and changed my code to </p>

<pre><code>t11 = soup.findAll("p", {"class": "commentsParagraph2"})
</code></pre>

<p>but t11 is an empty list.</p>

<p>I am attaching the page file as well<a href="https://justpaste.it/z64t" rel="nofollow">here</a></p>

<p>Any ideas?</p>

<p>I have downloaded you html and done some tests, the beautifulsoup module could only find out three p nodes. And I guess, that's because there is some iframes in the html, so BS does not work probably. My suggestion is using <code>re</code> module instead of <code>bs</code></p>

<p>sample code for your reference:</p>

<pre><code>import re

with open('1.html', 'r') as f:
    data = f.read()
    m=re.findall(r'(?&lt;=&lt;p class="commentsParagraph"&gt;)[\!\w\s.\'\,\-\(\)\@\#\$\%\^\&amp;\*\+\=\/|\^&lt;]+(?=&lt;/p&gt;)', data)
    print(m)
</code></pre>

python beautifulsoup