Python Web-scraping Solution
<p>So, I'm new to python and am trying to develop an exercise in which I scrape the page numbers from a list on this url, which is a list of various published papers. </p>

<p>When I go into the HTML element for the page I want to scrape, I inspect the element and find this HTML code to match up:</p>

<pre><code>&lt;div class="src"&gt;
        Foreign Affairs, Vol. 79, No. 4 (Jul. - Aug., 2000), pp. 53-63
    &lt;/div&gt;
</code></pre>

<p>The part that I want to churn out what is in between the class brackets.
This is what I attempted to write in order to do the job. </p>

<pre><code>import requests
from bs4 import BeautifulSoup

url = "http://www.jstor.org/action/doAdvancedSearch?c4=AND&amp;c5=AND&amp;q2=&amp;pt=&amp;q1=nuclear&amp;f3=all&amp;f1=all&amp;c3=AND&amp;c6=AND&amp;q6=&amp;f4=all&amp;q4=&amp;f0=all&amp;c2=AND&amp;q3=&amp;acc=off&amp;c1=AND&amp;isbn=&amp;q0=china+&amp;f6=all&amp;la=&amp;f2=all&amp;ed=2001&amp;q5=&amp;f5=all&amp;group=none&amp;sd=2000"
r = requests.get(url)
soup = BeautifulSoup(r.content)
links = soup.find_all("div class='src'")
for link in links:
    print 
</code></pre>

<p>I know that this code is unfinished and that's because I don't know where to go from here :/. Can anyone help me here?</p>

<p>An alternative to Tales PÃ¡dua's <a href="http://stackoverflow.com/a/39539709/189134">answer</a> is this:</p>

<pre><code>from bs4 import BeautifulSoup

html = """&lt;div class="src"&gt;
    Foreign Affairs, Vol. 79, No. 4 (Jul. - Aug., 2000), pp. 53-63
&lt;/div&gt;
&lt;div class="src"&gt;
    Other Book, Vol. 1, No. 1 (Jul. - Aug., 2000), pp. 1-23
&lt;/div&gt;"""
soup = BeautifulSoup(html)
links = soup.find_all("div", class_ = "src")
for link in links:
    print link.text.strip()
</code></pre>

<p>This outputs: </p>

<pre><code>Foreign Affairs, Vol. 79, No. 4 (Jul. - Aug., 2000), pp. 53-63
Other Book, Vol. 1, No. 1 (Jul. - Aug., 2000), pp. 1-23
</code></pre>

<p>This answer uses the <code>class_</code> parameter, which is <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-by-css-class" rel="nofollow">recommended</a> in the documentation. </p>

<hr>

<p>If you are looking to get the page number, and everything follows the format above (comma separated), you can change the for loop to grab the last element of the string:</p>

<pre><code>print link.text.split(",")[-1].strip()
</code></pre>

<p>This outputs:</p>

<pre><code>pp. 53-63
pp. 1-23
</code></pre>

python web-scraping beautifulsoup