How to scrape 'Click to Display' fields with BeautifulSoup
<p>I am trying to scrape the number of schools and names of schools that basketball players get offers from verbalcommits.com</p>

<p>Using this page as an example: <a href="http://www.verbalcommits.com/players/jarrey-foster" rel="nofollow">http://www.verbalcommits.com/players/jarrey-foster</a></p>

<p>It's easy to access the first offer (SMU) but all of the other offers are hidden behind the "Show other offers" button.  When I inspect the page, I can see the offers but my scraper doesn't get to them.  I've been using the following:</p>

<pre><code>page=urllib.request.urlopen("http://www.verbalcommits.com/players/jarrey-foster") #opens page    
soup = BeautifulSoup(page, 'html.parser') #makes page into a BS python object
schools = soup.body.findAll('span',{"class":"team_name"})
print(schools)
</code></pre>

<p>This returns the first span that has the team name in it, but not the rest of the spans that are hidden.  What do I need to add to access the rest of the page that is hidden?</p>

<p>You can't get other data because when you click button then JavaScript reads it from server from </p>

<p><a href="http://www.verbalcommits.com/player_divs/closed_offers?player_id=17766&amp;_=1475626846752" rel="nofollow">http://www.verbalcommits.com/player_divs/closed_offers?player_id=17766&amp;_=1475626846752</a></p>

<p>Now you can use this url with BS to get data.</p>

<p>I used <code>Firebug</code> in Firefox or <code>Developer Tools</code> in Chrome to find this url.</p>

<hr>

<p><strong>EDIT:</strong> inside HTML I found <code>data-player-id="17766"</code> - it is first argument in above url. Maybe you can find second argument so you could generate url using Python.</p>

<hr>

<p><strong>EDIT:</strong> I checked url </p>

<p><a href="http://www.verbalcommits.com/player_divs/closed_offers?player_id=17766" rel="nofollow">http://www.verbalcommits.com/player_divs/closed_offers?player_id=17766</a> </p>

<p>and it gives the same data so you don't need second argument.</p>

python html web-scraping beautifulsoup