Parse only text within a div class python
<p>so what I wanna do is from reading the source code, search for the div class named "gsc_prf_il", then within this div class, extract only the text, ignoring the href link. e.g. </p>

<pre><code>&lt;div class="gsc_prf_il"&gt;&lt;a href="/citations?view_op=view_org&amp;hl=en&amp;org=13784427342582529234"&gt;McGill University&lt;/a&gt;&lt;/div&gt;
</code></pre>

<p>But when I use this code, it doesn't work, only gives me error: AttributeError: 'NoneType' object has no attribute 'contents'</p>

<pre><code>soup=BeautifulSoup(p.readlines()[0], 'html.parser')
s=soup.find(id='gsc_prf_il')
scholar_info['department']= s.contents
</code></pre>

<p>So then I tried this: </p>

<pre><code>scholar_info['department']=[s.find('a')['href'], s.find('a').contents[0]]
</code></pre>

<p>It doesn't work either. What am I doing wrong? </p>

<p>Just find the <em>div</em> and pull the text, you are looking for <code>soup.find(id='gsc_prf_il')</code> which is looking for an element with an <code>id</code> of <code>gsc_prf_il</code> not a div with that class:</p>

<pre><code>from bs4 import BeautifulSoup
url = "http://python-data.dr-chuck.net/comments_283660.html"

soup = BeautifulSoup("""&lt;div class="gsc_prf_il"&gt;&lt;a href="/citations?view_op=view_org&amp;hl=en&amp;org=13784427342582529234"&gt;McGill University&lt;/a&gt;&lt;/div&gt;""")
</code></pre>

<p>So use <code>class_="gsc_prf_il"</code>:</p>

<pre><code>print(soup.find("div", class_="gsc_prf_il").text) -&gt; McGill University
</code></pre>

<p>Or use a css selector:</p>

<pre><code>print(soup.select_one("div.gsc_prf_il").text) -&gt; McGill University
</code></pre>

python python-2.7 beautifulsoup html-parsing