Extract data from &quote under title tag using BeautifulSoup?
<p>I want to extract title of a link after getting its HTML via <code>BeautifulSoup</code> library in python.
Basically, the whole title tag is </p>

<pre><code> &lt;title&gt;Imaan Z Hazir on Twitter: &amp;quot;Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)&amp;quot;&lt;/title&gt;
</code></pre>

<p>I want to extract data that is in &amp;quot tags that is only this <code>Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)</code>
I tried as </p>

<pre><code>import urllib
import urllib.request

from bs4 import BeautifulSoup

link = "https://twitter.com/ImaanZHazir/status/778560899061780481"
try:
    List=list()
    r = urllib.request.Request(link, headers={'User-Agent': 'Chrome/51.0.2704.103'})
    h = urllib.request.urlopen(r).read()
    data = BeautifulSoup(h,"html.parser")
    for i in data.find_all("title"):
        List.append(i.text)
        print(List[0])
except urllib.error.HTTPError as err:
    pass
</code></pre>

<p>I also tried as </p>

<pre><code>for i in data.find_all("title.&amp;quot"):

for i in data.find_all("title&gt;&amp;quot"):

for i in data.find_all("&amp;quot"):
</code></pre>

<p>and </p>

<pre><code>for i in data.find_all("quot"):
</code></pre>

<p>But no one is working.</p>

<p>Once you have parsed the html:</p>

<pre><code>data = BeautifulSoup(h,"html.parser")
</code></pre>

<p>Find the title this way:</p>

<pre><code>title = data.find("title").string  # this is without &lt;title&gt; tag
</code></pre>

<p>Now find two quotes (<code>"</code>) in the string. There are many ways to do that. I would use regex:</p>

<pre><code>import re
match = re.search(r'".*"', title)
if match:
    print match.group(0)
</code></pre>

<p>You never search for <code>&amp;quot;</code> or any other <code>&amp;NAME;</code> sequences because BeautifulSoup converts them to the actual characters they represent.</p>

<p><strong>EDIT:</strong></p>

<p>Regex which does not capture the quotes would be:</p>

<pre><code>re.search(r'(?&lt;=").*(?=")', title)
</code></pre>

python css-selectors beautifulsoup html-parser