Extract data from &quote under title tag using BeautifulSoup?
<p>I want to extract title of a link after getting its HTML via <code>BeautifulSoup</code> library in python.
Basically, the whole title tag is </p>

<pre><code> &lt;title&gt;Imaan Z Hazir on Twitter: &amp;quot;Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)&amp;quot;&lt;/title&gt;
</code></pre>

<p>I want to extract data that is in &amp;quot tags that is only this <code>Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)</code>
I tried as </p>

<pre><code>import urllib
import urllib.request

from bs4 import BeautifulSoup

link = "https://twitter.com/ImaanZHazir/status/778560899061780481"
try:
    List=list()
    r = urllib.request.Request(link, headers={'User-Agent': 'Chrome/51.0.2704.103'})
    h = urllib.request.urlopen(r).read()
    data = BeautifulSoup(h,"html.parser")
    for i in data.find_all("title"):
        List.append(i.text)
        print(List[0])
except urllib.error.HTTPError as err:
    pass
</code></pre>

<p>I also tried as </p>

<pre><code>for i in data.find_all("title.&amp;quot"):

for i in data.find_all("title&gt;&amp;quot"):

for i in data.find_all("&amp;quot"):
</code></pre>

<p>and </p>

<pre><code>for i in data.find_all("quot"):
</code></pre>

<p>But no one is working.</p>

<p>Here is a simple complete example using regex to extract the text within quotes:</p>

<pre><code>import urllib
import re
from bs4 import BeautifulSoup

link = "https://twitter.com/ImaanZHazir/status/778560899061780481"

r = urllib.request.urlopen(link)
soup = BeautifulSoup(r, "html.parser")
title = soup.title.string
quote = re.match(r'^.*\"(.*)\"', title)
print(quote.group(1))
</code></pre>

<p>What happen's here is that after getting pages's source and finding the <code>title</code> we use a regular expression against the title to extract the text within the quotes.</p>

<p>We tell the regular expression to look for an arbitrary number of symbols at the beginning of the string (<code>^.*</code>) before the opening quote (<code>\"</code>), then capture the text between it and the closing quote (second <code>\"</code>).</p>

<p>Then we print the captured text by telling Python to print the first captured group (the part between parenthesis in regex).</p>

<p>Here's more on matching with regex in python - <a href="https://docs.python.org/3/library/re.html#match-objects" rel="nofollow">https://docs.python.org/3/library/re.html#match-objects</a></p>

python css-selectors beautifulsoup html-parser