Beautiful Soup has extra </body> before actual end
<p>I am trying to scrape poems from PoetryFoundation.org. I have found in one of my test cases that when I pull the html from a specific poem it includes an extra <code>&lt;/body&gt;</code> before the end of the actual poem. I can look at the source code for the poem online and there is no  in the middle of the poem (as to be expected). I created an example with the url of a specific case such that others can try to replicate the problem:</p>

<pre><code>from bs4 import BeautifulSoup
from urllib.request import urlopen

poem_page = urlopen("https://www.poetryfoundation.org/poems-and-poets/poems/detail/57956")
poem_soup = BeautifulSoup(poem_page.read(), "html5lib")
print(poem_soup)
</code></pre>

<p>I'm running Python 3.5.1. I've tried this with the default parsers <code>html.parser</code> as well as <code>html5lib</code> and <code>lxml</code>.</p>

<p>In the print out, if you search for 'in the poem' you'll find this snippet of html, which makes no sense because it ends the entire html document midway through the poem with <code>&lt;/body&gt;&lt;/html&gt;</code> and then continues on with the rest of document:</p>

<pre><code>in the poem&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/div&gt;&lt;/body&gt;&lt;/html&gt;. But when we met,&lt;br/&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;&lt;br/&gt;
</code></pre>

<p>I've looked at the source code online and this is what it should be:</p>

<pre><code>in the poem&lt;/em&gt;. But when we met,&lt;br&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;
</code></pre>

<p>I have no idea why when I scrape it it's closing the entire html document partway through the page.</p>

<p>When I try to get the poem with your url with <code>html.parser</code>,I got the same problem as you.The html was truncated at the <code>in the poem</code> position.</p>

<pre><code>import requests
from bs4 import BeautifulSoup

poem_page = requests.get("https://www.poetryfoundation.org/poems-and-poets/poems/detail/57956")
poem_soup = BeautifulSoup(poem_page.text, "html.parser")
poem_div = poem_soup.find('div', class_='poem')
print poem_div
</code></pre>

<p>OUTPUT:</p>

<pre><code>&lt;div class="poem" data-view="ContentView"&gt;
&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;It seems a certain fear underlies everything. &lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;If I were to tell you something profound&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt; it would be useless, as every single thing I know&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt; is not timeless. I am particularly risk-averse.&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;I choose someone else over me every time, &lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;as I'm sure they'll finish the task at hand, &lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;which is to say that whatever is in front of us&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt; will get done if I'm not in charge of it.&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;&lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;There is a limit to the number of times &lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;I can practice every single kind of mortification &lt;br/&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;(of the flesh?). I can turn toward you and say &lt;em&gt;yes, &lt;br/&gt;&lt;/em&gt;&lt;/div&gt;&lt;div style="text-indent: -1em; padding-left: 1em;"&gt;it was you in the poem&lt;/div&gt;&lt;/div&gt;
</code></pre>

<p>But changing the parser to <code>lxml</code>,everything is ok.</p>

<pre><code>import requests
from bs4 import BeautifulSoup

poem_page = requests.get("https://www.poetryfoundation.org/poems-and-poets/poems/detail/57956")
poem_soup = BeautifulSoup(poem_page.text, "lxml")
poem_div = poem_soup.find('div', class_='poem')
# print poem_div
for s in poem_div.find_all('div'):
    print list(s.children)[0]
</code></pre>

<p>OUTPUT:</p>

<pre><code>It seems a certain fear underlies everything. 
If I were to tell you something profound
 it would be useless, as every single thing I know
 is not timeless. I am particularly risk-averse.
&lt;br/&gt;
I choose someone else over me every time, 
as I'm sure they'll finish the task at hand, 
which is to say that whatever is in front of us
 will get done if I'm not in charge of it.
&lt;br/&gt;
There is a limit to the number of times 
I can practice every single kind of mortification 
(of the flesh?). I can turn toward you and say 
it was you in the poem. But when we met,
&lt;br/&gt;
you were actually wearing a shirt, and the poem 
wasn't about you or your indecipherable tattoo. 
The poem is always about me, but that one time 
I was in love with the memory of my twenties
&lt;br/&gt;
so I was, for a moment, in love with you 
because you remind me of an approaching
 subway brushing hair off my face with 
its hot breath. Darkness. And then light,
&lt;br/&gt;
the exact goldness of dawn fingering
 that brick wall out my bedroom window 
on Smith Street mornings when I'd wake
 next to godknowswho but always someone
&lt;br/&gt;
who wasn't a mistake, because what kind 
of mistakes are that twitchy and joyful 
even if they're woven with a particular 
thread of regret: the guy who used
&lt;br/&gt;
my toothbrush without asking,
I walked to the end of a pier with him,
would have walked off anywhere with him
until one day we both landed in California
&lt;br/&gt;
when I was still young, and going West
meant taking a laptop and some clothes
in a hatchback and learning about produce.
I can turn toward you, whoever you are,
&lt;br/&gt;
and say you are my lover simply because
I say you are, and that is, I realize,
a tautology, but this is my poem. I claim
nothing other than what I write, and even that,
&lt;br/&gt;
I'd leave by the wayside, since the only thing
to pack would be the candlesticks, and 
even those are burned through, thoroughly
replaceable. Who am I kidding? I don't
&lt;br/&gt;
own anything worth packing into anything.
We are cardboard boxes, you and I, stacked
nowhere near each other and humming
different tunes. It is too late to be writing this.
&lt;br/&gt;
I am writing this to tell you something less
than neutral, which is to say I'm sorry.
It was never you. It was always you:
your unutterable name, this growl in my throat.
&lt;br/&gt;
</code></pre>

python html web-scraping beautifulsoup