BeautifulSoup - How do I get all <div> from a class in html?
<p>I am trying to get a list of all NFL teams from a website and I am very close. I am able to get some data, but I can't drill down far enough to get what I want.</p>

<p>My code:</p>

<pre><code>from bs4 import BeautifulSoup
import requests

f = open('C:\Users\Josh\Documents\Python\outFileRoto.txt', 'w')
errorFile = open('C:\Users\Josh\Documents\Python\errors.txt', 'w')



r  = requests.get('https://rotogrinders.com/team-stats/nfl-allowed?sport=nfl&amp;position=QB&amp;site=draftkings&amp;range=season')
data = r.text
#soup = BeautifulSoup(urllib2.urlopen('http://games.espn.com/ffl/tools/projections?startIndex=' +str(x).read(), 'html')
soup = BeautifulSoup(data, 'html.parser')
leftTable = soup.find('div', attrs={'class' : 'rgt-bdy left'})
#f.write("LEFT TABLE\n" + str(leftTable) + '\n')

rightCol = leftTable.find('div', attrs={'class' : 'rgt-colwrap'})

for row in rightCol.findAll('div'):
    #col = row.findAll('div')
    #f.write("col" + str(col))

    try:
        name = str(row)
        f.write("----------------------------COLUMN---------------------------\n" + name + '\n')

    except Exception as e:
        errorFile.write (str(x) + "&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;" + str(a) + "&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;ROW" + str(row) + '\n')
        pass


f.close
errorFile.close
</code></pre>

<p>The problem is that I get this:</p>

<pre><code>----------------------------COLUMN---------------------------
&lt;div class="rgt-col"&gt;
&lt;div class="rgt-hdr"&gt;Team&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
----------------------------COLUMN---------------------------
&lt;div class="rgt-hdr"&gt;Team&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
----------------------------COLUMN---------------------------
&lt;div class="rgt-col"&gt;
&lt;div class="rgt-hdr"&gt;Abbr&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
----------------------------COLUMN---------------------------
&lt;div class="rgt-hdr"&gt;Abbr&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
</code></pre>

<p>But I need this:</p>

<p><img src="http://i.stack.imgur.com/gPuXf.png" alt="NFL Teams"></p>

<p>To drill down further, get beautifulsoup to return the div that has the class "rgt-col", and the style "display: block;".</p>

<p>Once you have that, drill down further by finding all the divs within that div, but ignoring the first result. Or you can also get all the divs that do not have a class.</p>

<p>EDIT 1:
This answer was provided with the assumption that the html code was already available, and all that was needed was to drill down to get the specific elements.  However, as mentioned by the Padraic Cunningham and Casey wireman, the desired data is dynamically loaded, and as such, the html is not available in the first place.  Therefore, the first step would be to obtain the html first, maybe through identifying and loading the json endpoint, or, through the use of tools which allow for browser automation such as selenium.</p>

<p>EDIT 2:
In this case however, it seems that the desired data is already in the html, in json format.  All that's left is to parse this as was done by Padraic Cunningham in his answer.</p>

python python-2.7 beautifulsoup python-requests