BeautifulSoup - How do I get all <div> from a class in html?
<p>I am trying to get a list of all NFL teams from a website and I am very close. I am able to get some data, but I can't drill down far enough to get what I want.</p>

<p>My code:</p>

<pre><code>from bs4 import BeautifulSoup
import requests

f = open('C:\Users\Josh\Documents\Python\outFileRoto.txt', 'w')
errorFile = open('C:\Users\Josh\Documents\Python\errors.txt', 'w')



r  = requests.get('https://rotogrinders.com/team-stats/nfl-allowed?sport=nfl&amp;position=QB&amp;site=draftkings&amp;range=season')
data = r.text
#soup = BeautifulSoup(urllib2.urlopen('http://games.espn.com/ffl/tools/projections?startIndex=' +str(x).read(), 'html')
soup = BeautifulSoup(data, 'html.parser')
leftTable = soup.find('div', attrs={'class' : 'rgt-bdy left'})
#f.write("LEFT TABLE\n" + str(leftTable) + '\n')

rightCol = leftTable.find('div', attrs={'class' : 'rgt-colwrap'})

for row in rightCol.findAll('div'):
    #col = row.findAll('div')
    #f.write("col" + str(col))

    try:
        name = str(row)
        f.write("----------------------------COLUMN---------------------------\n" + name + '\n')

    except Exception as e:
        errorFile.write (str(x) + "&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;" + str(a) + "&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;ROW" + str(row) + '\n')
        pass


f.close
errorFile.close
</code></pre>

<p>The problem is that I get this:</p>

<pre><code>----------------------------COLUMN---------------------------
&lt;div class="rgt-col"&gt;
&lt;div class="rgt-hdr"&gt;Team&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
----------------------------COLUMN---------------------------
&lt;div class="rgt-hdr"&gt;Team&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
----------------------------COLUMN---------------------------
&lt;div class="rgt-col"&gt;
&lt;div class="rgt-hdr"&gt;Abbr&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
&lt;/div&gt;
----------------------------COLUMN---------------------------
&lt;div class="rgt-hdr"&gt;Abbr&lt;span class="icn-arw-down"&gt;&lt;/span&gt;&lt;/div&gt;
</code></pre>

<p>But I need this:</p>

<p><img src="http://i.stack.imgur.com/gPuXf.png" alt="NFL Teams"></p>

<p>It looks like your main problem is that the table you're interested in is dynamically built by JS.  See this answer for info on scraping dynamically loaded content.  <a href="http://stackoverflow.com/questions/17597424/how-to-retrieve-the-values-of-dynamic-html-content-using-python">How to retrieve the values of dynamic html content using Python</a></p>

<p>Alternatively, it looks like they have the initialization data that they generate the table with in the page, you could scrape that and parse the array that way if you don't want to go to the trouble of setting up things like selenium.</p>

python python-2.7 beautifulsoup python-requests