How to scrape 'Click to Display' fields with BeautifulSoup
<p>I am trying to scrape the number of schools and names of schools that basketball players get offers from verbalcommits.com</p>

<p>Using this page as an example: <a href="http://www.verbalcommits.com/players/jarrey-foster" rel="nofollow">http://www.verbalcommits.com/players/jarrey-foster</a></p>

<p>It's easy to access the first offer (SMU) but all of the other offers are hidden behind the "Show other offers" button.  When I inspect the page, I can see the offers but my scraper doesn't get to them.  I've been using the following:</p>

<pre><code>page=urllib.request.urlopen("http://www.verbalcommits.com/players/jarrey-foster") #opens page    
soup = BeautifulSoup(page, 'html.parser') #makes page into a BS python object
schools = soup.body.findAll('span',{"class":"team_name"})
print(schools)
</code></pre>

<p>This returns the first span that has the team name in it, but not the rest of the spans that are hidden.  What do I need to add to access the rest of the page that is hidden?</p>

<p>To elaborate more on <a href="http://stackoverflow.com/a/39863674/771848">@furas's great answer</a>. Here is how you can extract the player id and make a second request to get the "closed offers". For this, we are going to <a class='doc-link' href="http://stackoverflow.com/documentation/python/1792/web-scraping-with-python/8152/maintaining-web-scraping-session-with-requests#t=201610051326031908792">maintain a web-scraping session with <code>requests</code></a>:</p>

<pre><code>import requests
from bs4 import BeautifulSoup


with requests.Session() as session:
    response = session.get("http://www.verbalcommits.com/players/jarrey-foster")

    # get the player id
    soup = BeautifulSoup(response.content, "html.parser")
    player_id = soup.select_one("h1.player-name").get("data-player-id")

    # get closed offers
    response = session.get("http://www.verbalcommits.com/player_divs/closed_offers", params={"player_id": player_id})

    soup = BeautifulSoup(response.content, "html.parser")

    # print team names
    for team in soup.select(".team_name"):
        print(team.get_text())
</code></pre>

<p>Prints team names for demonstration purposes:</p>

<pre><code>UTEP
Sam Houston State
New Hampshire
Rice
Temple
Liberty
UL Lafayette
</code></pre>

python html web-scraping beautifulsoup