Python: (Beautifulsoup) How to limit extracted text from a html news article to only the news article.
<p>I wrote this test code which uses BeautifulSoup.</p>

<pre><code>url = "http://www.dailymail.co.uk/news/article-3795511/Harry-Potter-sale-half-million-pound-house-Iconic-Privet-Drive-market-suburban-Berkshire-complete-cupboard-stairs-one-magical-boy.html"    
html = urllib.request.urlopen(url).read()  
soup = BeautifulSoup(html,"lxml")
for n in soup.find_all('p'):
    print(n.get_text())
</code></pre>

<p>It works fine but it also retrieves text that is not part of the news article, such as the time it was posted, number of comments, copyrights ect. </p>

<p>I would wish for it to only retrieve text from the news article itself, how would one go about this? </p>

<p>You might have much better luck with <a href="http://newspaper.readthedocs.io/en/latest/" rel="nofollow"><code>newspaper</code> library</a> which is focused on scraping articles.</p>

<p>If we talk about <code>BeautifulSoup</code> only, one option to get closer to the desired result and have more relevant paragraphs is to find them in the context of <code>div</code> element with <code>itemprop="articleBody"</code> attribute:</p>

<pre><code>article_body = soup.find(itemprop="articleBody")
for p in article_body.find_all("p"):
    print(p.get_text())
</code></pre>

python html beautifulsoup