/n in beautiful soup text
<p>I'm trying to get the transcript for a youtube video for some NLP work, and I think I can get it fine, but there are also a couple of issues. For example:   </p>

<pre><code>from xml.etree import cElementTree as ET
from bs4 import BeautifulSoup as bs
from urllib2 import urlopen

URL = 'http://video.google.com/timedtext?lang=en&amp;v=KDHuWxy53uM'
def make_soup(url):
    html = urlopen(url).read()
    return bs(html, "lxml")

soup = make_soup(URL)
takeaways = soup.findAll('text')

All_text = []
for i in takeaways:
    root = ET.fromstring(str(i))
    reslist = list(root.iter())
    try:
        result = ' '.join([element.text for element in reslist])
    except:
        pass
    All_text.append(result)
</code></pre>

<p>Sample result for one of the lines:</p>

<pre><code>'Let&amp;#39;s learn a little bit\nabout the dot product.'
</code></pre>

<p>This seems ok to get the transcript, but I also get the /n which are the return characters from the xml and I also get this weird character in place of the apostrophes, which I think is due to encoding?</p>

<p>Anybody know how I can clean both of these up? As always, thank you all so much for any help you can give.</p>

<p>&amp;#39 - is unescaped HTML code , in  <strong>python 3.2</strong> and above use </p>

<pre><code>import html
html.unescape(&lt;your_string&gt;) 
</code></pre>

python beautifulsoup