&pound; displaying in urllib2 and Beautiful Soup
<p>I'm trying to write a small web scraper in python, and I think I've run into an encoding issue. I'm trying to scrape <a href="http://www.resident-music.com/tickets" rel="nofollow">http://www.resident-music.com/tickets</a> (specifically the table on the page) - a row might look something like this -</p>

<pre><code>    &lt;tr&gt;
        &lt;td style="width:64.9%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;the great escape 2017&amp;nbsp; local early bird tickets, selling fast&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:13.1%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;18&lt;sup&gt;th&lt;/sup&gt;&amp;ndash; 20&lt;sup&gt;th&lt;/sup&gt; may&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:15.42%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;various&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:6.58%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;&amp;pound;55.00&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
       &lt;/tr&gt;
</code></pre>

<p>I'm essentially trying to replace the <code>&amp;pound;55.00</code> with Â£55, and any other 'non-text' nasties.</p>

<p>I've tried a few different encoding things you can go with beautifulsoup, and urllib2 - to no avail, I think I'm just doing it all wrong.</p>

<p>Thanks</p>

<p>You want to <em>unescape</em> the html which you can do using <em>html.unescape</em> in python3:</p>

<pre><code>In [14]: from html import unescape

In [15]: h = """&lt;tr&gt;
   ....:         &lt;td style="width:64.9%;height:11px;"&gt;
   ....:          &lt;p&gt;&lt;strong&gt;the great escape 2017&amp;nbsp; local early bird tickets, selling fast&lt;/strong&gt;&lt;/p&gt;
   ....:         &lt;/td&gt;
   ....:         &lt;td style="width:13.1%;height:11px;"&gt;
   ....:          &lt;p&gt;&lt;strong&gt;18&lt;sup&gt;th&lt;/sup&gt;&amp;ndash; 20&lt;sup&gt;th&lt;/sup&gt; may&lt;/strong&gt;&lt;/p&gt;
   ....:         &lt;/td&gt;
   ....:         &lt;td style="width:15.42%;height:11px;"&gt;
   ....:          &lt;p&gt;&lt;strong&gt;various&lt;/strong&gt;&lt;/p&gt;
   ....:         &lt;/td&gt;
   ....:         &lt;td style="width:6.58%;height:11px;"&gt;
   ....:          &lt;p&gt;&lt;strong&gt;&amp;pound;55.00&lt;/strong&gt;&lt;/p&gt;
   ....:         &lt;/td&gt;
   ....:        &lt;/tr&gt;"""

In [16]: 

In [16]: print(unescape(h))
&lt;tr&gt;
        &lt;td style="width:64.9%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;the great escape 2017Â  local early bird tickets, selling fast&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:13.1%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;18&lt;sup&gt;th&lt;/sup&gt;â 20&lt;sup&gt;th&lt;/sup&gt; may&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:15.42%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;various&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:6.58%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;Â£55.00&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
       &lt;/tr&gt;
</code></pre>

<p>For python2 use:</p>

<pre><code>In [6]: from html.parser import HTMLParser

In [7]: unescape = HTMLParser().unescape  

In [8]: print(unescape(h))
&lt;tr&gt;
        &lt;td style="width:64.9%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;the great escape 2017Â  local early bird tickets, selling fast&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:13.1%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;18&lt;sup&gt;th&lt;/sup&gt;â 20&lt;sup&gt;th&lt;/sup&gt; may&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:15.42%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;various&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
        &lt;td style="width:6.58%;height:11px;"&gt;
         &lt;p&gt;&lt;strong&gt;Â£55.00&lt;/strong&gt;&lt;/p&gt;
        &lt;/td&gt;
</code></pre>

<p>You can see both correctly unescape all entities not just the pound sign.</p>

python encoding beautifulsoup urllib2