loop stuck on first page
<p>Been using beautiful soup to iterate through pages, but for whatever reason I can't get the loop to advance beyond the first page.  it seems like it should be easy because it's a text string, but it seems to loop back, maybe it's my structure not my text string?</p>

<p>Here's what I have:</p>

<pre><code>import csv
import urllib2
from bs4 import BeautifulSoup

f = open('nhlstats.csv', "w")


groups=['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties']

year = ["2016", "2015","2014","2013","2012"]

for yr in year:
    for gr in groups:
        url = "http://www.espn.com/nhl/statistics/player/_/stat/points/year/"+str(yr)
    #www.espn.com/nhl/statistics/player/_/stat/points/year/2014/
    page = urllib2.urlopen(url)
    soup=BeautifulSoup(page, "html.parser")
    pagecount = soup.findAll(attrs= {"class":"page-numbers"})[0].string
    pageliteral = int(pagecount[5:])
    for i in range(0,pageliteral):
        number = int(((i*40) + 1))
        URL = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/"+str(yr) + "/count/"+str(number)
        page = urllib2.urlopen(url)
        soup=BeautifulSoup(page, "html.parser")
        for tr in soup.select("#my-players-table tr[class*=player]"):
            row =[]
            for ob in range(1,15):
                player_info = tr('td')[ob].get_text(strip=True)
                row.append(player_info)
            f.write(str(yr) +","+",".join(row) + "\n")

f.close()
</code></pre>

<p>this gets the same first 40 records over and over.  </p>

<p>I tried using <a href="https://automatetheboringstuff.com/chapter11/" rel="nofollow">this solution</a> as an if and did find that doing </p>

<pre><code>prevLink = soup.select('a[rel="nofollow"]')[0]
newurl =  "http:" + prevLink.get('href')
</code></pre>

<p>did work better, but I'm not sure how to do the loop in such a way that it advances? possibly just tired but my loop there still just goes to the next set of records and gets stuck on <em>that</em> one. please help me fix my loop</p>

<p><strong>UPDATE</strong></p>

<p>my formatting was lost in the copy paste, my actual code looks like:</p>

<pre><code>import csv
import urllib2
from bs4 import BeautifulSoup

f = open('nhlstats.csv', "w")


groups=['points', 'shooting', 'goaltending', 'defensive', 'timeonice', 'faceoffs', 'minor-penalties', 'major-penalties']

year = ["2016", "2015","2014","2013","2012"]


for yr in year:
    for gr in groups:
        url = "http://www.espn.com/nhl/statistics/player/_/stat/points/year/"+str(yr)
    #www.espn.com/nhl/statistics/player/_/stat/points/year/2014/
        page = urllib2.urlopen(url)
        soup=BeautifulSoup(page, "html.parser")
        pagecount = soup.findAll(attrs= {"class":"page-numbers"})[0].string
        pageliteral = int(pagecount[5:])
        for i in range(0,pageliteral):
            number = int(((i*40) + 1))
            URL = "http://www.espn.com/nhl/statistics/player/_/stat/points/sort/points/year/"+str(yr) + "/count/"+str(number)
            page = urllib2.urlopen(url)
            soup=BeautifulSoup(page, "html.parser")
            for tr in soup.select("#my-players-table tr[class*=player]"):
                row =[]
                for ob in range(1,15):
                    player_info = tr('td')[ob].get_text(strip=True)
                    row.append(player_info)
                f.write(str(yr) +","+",".join(row) + "\n")

f.close()
</code></pre>

<p>You are changing the URL many times before you are opening it the first time, due to an indentation error. Try this:</p>

<p><code>for gr in groups:
        url = "...some_url..."
        page = urllib2.urlopen(url)
        ...everything else should be indented....</code></p>

python python-2.7 scripting beautifulsoup