Extract data from &quote under title tag using BeautifulSoup?
<p>I want to extract title of a link after getting its HTML via <code>BeautifulSoup</code> library in python.
Basically, the whole title tag is </p>

<pre><code> &lt;title&gt;Imaan Z Hazir on Twitter: &amp;quot;Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)&amp;quot;&lt;/title&gt;
</code></pre>

<p>I want to extract data that is in &amp;quot tags that is only this <code>Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)</code>
I tried as </p>

<pre><code>import urllib
import urllib.request

from bs4 import BeautifulSoup

link = "https://twitter.com/ImaanZHazir/status/778560899061780481"
try:
    List=list()
    r = urllib.request.Request(link, headers={'User-Agent': 'Chrome/51.0.2704.103'})
    h = urllib.request.urlopen(r).read()
    data = BeautifulSoup(h,"html.parser")
    for i in data.find_all("title"):
        List.append(i.text)
        print(List[0])
except urllib.error.HTTPError as err:
    pass
</code></pre>

<p>I also tried as </p>

<pre><code>for i in data.find_all("title.&amp;quot"):

for i in data.find_all("title&gt;&amp;quot"):

for i in data.find_all("&amp;quot"):
</code></pre>

<p>and </p>

<pre><code>for i in data.find_all("quot"):
</code></pre>

<p>But no one is working.</p>

<p>Just split the text on the colon:</p>

<pre><code>In [1]:  h = """&lt;title&gt;Imaan Z Hazir on Twitter: &amp;quot;Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)&amp;quot;&lt;/title&gt;"""

In [2]: from bs4 import BeautifulSoup

In [3]: soup  = BeautifulSoup(h, "lxml")

In [4]: print(soup.title.text.split(": ", 1)[1])
 "Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)"
</code></pre>

<p>Actually looking at the page you don't need to split at all, the text is in the <em>p</em> tag inside the <em>div.js-tweet-text-container</em>, th:</p>

<pre><code>In [8]: import requests

In [9]: from bs4 import BeautifulSoup


In [10]: soup  = BeautifulSoup(requests.get("https://twitter.com/ImaanZHazir/status/778560899061780481").content, "lxml")


In [11]: print(soup.select_one("div.js-tweet-text-container p").text)
Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)

In [12]: print(soup.title.text.split(": ", 1)[1])
"Guantanamo and Abu Ghraib, financial and military support to dictators in Latin America during the cold war. REALLY, AMERICA? (3)"
</code></pre>

<p>So you can do it either way for the same result.</p>

python css-selectors beautifulsoup html-parser