loop stuck on first page
<p>Been using beautiful soup to iterate through pages, but for whatever reason I can't get the loop to advance beyond the first page.  it seems like it should be easy because it's a text string, but it seems to loop back, maybe it's my structure not my text string?</p>

<p>Here's what I have:</p>

<pre><code>import csv
import urllib2
from bs4 import BeautifulSoup

f = open('nhlstats.csv', "w")


groups=['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties']

year = ["2016", "2015","2014","2013","2012"]

for yr in year:
    for gr in groups:
        url = "http://www.espn.com/nhl/statistics/player/_/stat/points/year/"+str(yr)
    #www.espn.com/nhl/statistics/player/_/stat/points/year/2014/
    page = urllib2.urlopen(url)
    soup=BeautifulSoup(page, "html.parser")
    pagecount = soup.findAll(attrs= {"class":"page-numbers"})[0].string
    pageliteral = int(pagecount[5:])
    for i in range(0,pageliteral):
        number = int(((i*40) + 1))
        URL = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/"+str(yr) + "/count/"+str(number)
        page = urllib2.urlopen(url)
        soup=BeautifulSoup(page, "html.parser")
        for tr in soup.select("#my-players-table tr[class*=player]"):
            row =[]
            for ob in range(1,15):
                player_info = tr('td')[ob].get_text(strip=True)
                row.append(player_info)
            f.write(str(yr) +","+",".join(row) + "\n")

f.close()
</code></pre>

<p>this gets the same first 40 records over and over.  </p>

<p>I tried using <a href="https://automatetheboringstuff.com/chapter11/" rel="nofollow">this solution</a> as an if and did find that doing </p>

<pre><code>prevLink = soup.select('a[rel="nofollow"]')[0]
newurl =  "http:" + prevLink.get('href')
</code></pre>

<p>did work better, but I'm not sure how to do the loop in such a way that it advances? possibly just tired but my loop there still just goes to the next set of records and gets stuck on <em>that</em> one. please help me fix my loop</p>

<p><strong>UPDATE</strong></p>

<p>my formatting was lost in the copy paste, my actual code looks like:</p>

<pre><code>import csv
import urllib2
from bs4 import BeautifulSoup

f = open('nhlstats.csv', "w")


groups=['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties']

year = ["2016", "2015","2014","2013","2012"]


for yr in year:
    for gr in groups:
        url = "http://www.espn.com/nhl/statistics/player/_/stat/points/year/"+str(yr)
    #www.espn.com/nhl/statistics/player/_/stat/points/year/2014/
        page = urllib2.urlopen(url)
        soup=BeautifulSoup(page, "html.parser")
        pagecount = soup.findAll(attrs= {"class":"page-numbers"})[0].string
        pageliteral = int(pagecount[5:])
        for i in range(0,pageliteral):
            number = int(((i*40) + 1))
            URL = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/"+str(yr) + "/count/"+str(number)
            page = urllib2.urlopen(url)
            soup=BeautifulSoup(page, "html.parser")
            for tr in soup.select("#my-players-table tr[class*=player]"):
                row =[]
                for ob in range(1,15):
                    player_info = tr('td')[ob].get_text(strip=True)
                    row.append(player_info)
                f.write(str(yr) +","+",".join(row) + "\n")

f.close()
</code></pre>

<p>Your code indenting was mostly at fault. Also it would be wise to actually use the CSV library you imported, this will automatically wrap the player names in quotes to avoid any commas inside from ruining the csv structure.</p>

<p>This works by looking for the link to the next page and extracting the starting count. This is then used to build your the next page get. If no next page can be found, it moves to the next year group. Note, the count is not a page count but a starting entry count.</p>

<pre><code>import csv
import urllib2
from bs4 import BeautifulSoup


groups= ['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties']
year = ["2016", "2015", "2014", "2013", "2012"]

with open('nhlstats.csv', "wb") as f_output:
    csv_output = csv.writer(f_output)

    for yr in year:
        for gr in groups:
            start_count = 1
            while True:
                #print "{}, {}, {}".format(yr, gr, start_count)     # show progress

                url = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/{}/count/{}".format(yr, start_count)
                page = urllib2.urlopen(url)
                soup = BeautifulSoup(page, "html.parser")

                for tr in soup.select("#my-players-table tr[class*=player]"):
                    row = [yr]
                    for ob in range(1, 15):
                        player_info = tr('td')[ob].get_text(strip=True)
                        row.append(player_info)

                    csv_output.writerow(row)

                try:
                    start_count = int(soup.find(attrs= {"class":"page-numbers"}).find_next('a')['href'].rsplit('/', 1)[1])
                except:
                    break
</code></pre>

<p>Using <code>with</code> will also automatically close your file at the end.</p>

<p>This would give you a csv file starting as follows:</p>

<pre><code>2016,"Patrick Kane, RW",CHI,82,46,60,106,17,30,1.29,287,16.0,9,17,20
2016,"Jamie Benn, LW",DAL,82,41,48,89,7,64,1.09,247,16.6,5,17,13
2016,"Sidney Crosby, C",PIT,80,36,49,85,19,42,1.06,248,14.5,9,10,14
2016,"Joe Thornton, C",SJ,82,19,63,82,25,54,1.00,121,15.7,6,8,21
</code></pre>

python python-2.7 scripting beautifulsoup