Python Web-scraping Solution
<p>So, I'm new to python and am trying to develop an exercise in which I scrape the page numbers from a list on this url, which is a list of various published papers. </p>

<p>When I go into the HTML element for the page I want to scrape, I inspect the element and find this HTML code to match up:</p>

<pre><code>&lt;div class="src"&gt;
        Foreign Affairs, Vol. 79, No. 4 (Jul. - Aug., 2000), pp. 53-63
    &lt;/div&gt;
</code></pre>

<p>The part that I want to churn out what is in between the class brackets.
This is what I attempted to write in order to do the job. </p>

<pre><code>import requests
from bs4 import BeautifulSoup

url = "http://www.jstor.org/action/doAdvancedSearch?c4=AND&amp;c5=AND&amp;q2=&amp;pt=&amp;q1=nuclear&amp;f3=all&amp;f1=all&amp;c3=AND&amp;c6=AND&amp;q6=&amp;f4=all&amp;q4=&amp;f0=all&amp;c2=AND&amp;q3=&amp;acc=off&amp;c1=AND&amp;isbn=&amp;q0=china+&amp;f6=all&amp;la=&amp;f2=all&amp;ed=2001&amp;q5=&amp;f5=all&amp;group=none&amp;sd=2000"
r = requests.get(url)
soup = BeautifulSoup(r.content)
links = soup.find_all("div class='src'")
for link in links:
    print 
</code></pre>

<p>I know that this code is unfinished and that's because I don't know where to go from here :/. Can anyone help me here?</p>

<p>If I understand you correctly, you want the pages inside all divs with class="src"</p>

<p>If so, then you need to do:</p>

<pre><code>import requests
import re
from bs4 import BeautifulSoup

url = "http://www.jstor.org/action/doAdvancedSearch?c4=AND&amp;c5=AND&amp;q2=&amp;pt=&amp;q1=nuclear&amp;f3=all&amp;f1=all&amp;c3=AND&amp;c6=AND&amp;q6=&amp;f4=all&amp;q4=&amp;f0=all&amp;c2=AND&amp;q3=&amp;acc=off&amp;c1=AND&amp;isbn=&amp;q0=china+&amp;f6=all&amp;la=&amp;f2=all&amp;ed=2001&amp;q5=&amp;f5=all&amp;group=none&amp;sd=2000"
r = requests.get(url)
soup = BeautifulSoup(r.content)
links = soup.find_all('div', {'class':'src'})
for link in links:
    pages = re.search('(pp.\s*\d*-\d*)', link.text)
    print pages.group(1)
</code></pre>

<p>Note that I have used regex to get the page numbers. This may sound strange for people unfamiliar with regular expressions, but I think its more elegant than using string operations like <code>strip</code> and <code>split</code></p>

python web-scraping beautifulsoup