Navigating pagination with selenium
<p>I am getting stuck on a weird case of pagination.  I am scraping search results from <a href="https://cotthosting.com/NYRocklandExternal/LandRecords/protected/SrchQuickName.aspx" rel="nofollow">https://cotthosting.com/NYRocklandExternal/LandRecords/protected/SrchQuickName.aspx</a></p>

<p>I have search results that fall into 4 categories.</p>

<p>1) There are no search results</p>

<p>2) There is one results page</p>

<p>3) There is more than one results page but less than 12 results pages</p>

<p>4) There are more than 12 results pages.</p>

<p>For case 1, that is easy, I am just passing.</p>

<pre><code>results = driver.find_element_by_class_name('GridView')
if len(results)== 0:
    pass
</code></pre>

<p>For cases 2 and 3, I am checking if the list of links in the containing element is at least one and then click it.</p>

<pre><code>else:
    results_table = bsObj.find('table', {'class':'GridView'})
    sub_tables = results_table.find_all('table')
    next_page_links = sub_tables[1].find_all('a')
    if len(next_page_links) == 0
        scrapeResults()
    else:
        scrapeResults()
        ####GO TO NEXT PAGE UNTIL THERE IS NO NEXT PAGE
</code></pre>

<p>Question for case 2 and 3: What could i possibly check for here as my control? </p>

<p>The links are hrefs to pages 2, 3, etc. But the tricky part is if I am on a current page, say page 1, how do I make sure I a going to page 2 and when  I am on page 2 how do i make sure I am going to page 3? The html for page 1 for the results list is as follows</p>

<pre><code>&lt;table cellspacing="0" cellpadding="0" border="0" style="border-collapse:collapse;"&gt;
   &lt;tr&gt;
      &lt;td&gt;Page: &lt;span&gt;1&lt;/span&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$2&amp;#39;)"&gt;2&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$3&amp;#39;)"&gt;3&lt;/a&gt;&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;
</code></pre>

<p>I can zero into this table specifically using <code>sub_tables[1]</code> see above bs4 code in case 2.</p>

<p>The problem is there is no next button that I could utilize. Nothing changes along the results pages in the html. There is nothing to isolate the current page besides the number in the <code>span</code> right before the links. And I would like it to stop when it reaches the last page</p>

<p>For case 4, the html  looks like this:</p>

<pre><code>&lt;table cellspacing="0" cellpadding="0" border="0" style="border-collapse:collapse;"&gt;
   &lt;tr&gt;
      &lt;td&gt;Page: &lt;span&gt;1&lt;/span&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$2&amp;#39;)"&gt;2&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$3&amp;#39;)"&gt;3&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$4&amp;#39;)"&gt;4&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$5&amp;#39;)"&gt;5&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$6&amp;#39;)"&gt;6&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$7&amp;#39;)"&gt;7&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$8&amp;#39;)"&gt;8&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$9&amp;#39;)"&gt;9&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$10&amp;#39;)"&gt;10&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$11&amp;#39;)"&gt;...&lt;/a&gt;&lt;/td&gt;
      &lt;td&gt;&lt;a href="javascript:__doPostBack(&amp;#39;ctl00$cphMain$lrrgResults$cgvNamesDir&amp;#39;,&amp;#39;Page$Last&amp;#39;)"&gt;Last&lt;/a&gt;&lt;/td&gt;
   &lt;/tr&gt;
&lt;/table&gt;
</code></pre>

<p>The last two links are <code>...</code> to show that there are more results pages and <code>Last</code> to signify the last page. However, the `Last link exists on every page and it is only on the last page itself that it is not an active link. </p>

<p>Question for case 4, how could i check if the <code>last</code> link is clickable and use this as my stopping point? </p>

<p>Bigger question for case 4, how do i manouver the <code>...</code> to go through other results pages? The results page list is 12 values at most. i.e. the nearest ten pages to the current page, the <code>...</code> link to more pages and the <code>Last</code> link. So i don't know what to do if my results have say 88 pages.</p>

<p>I am link a dump to a full sample page : <a href="https://ghostbin.com/paste/nrb27" rel="nofollow">https://ghostbin.com/paste/nrb27</a></p>

<p>What you should do is to count the number of results in a page and use the value from total results to estimate the total number of pages by dividing.</p>

<p>If you will inspect the page you will see: `</p>

<p><code>Displaying records <strong>1 - 500</strong> of <strong>32563</strong> at <strong>10:08 AM ET</strong> on <strong>9/16/2016</strong></code></p>

<p>Knowing the total number of the page, start navigation and check that page is loaded if needed and knowing the current page you could get a dynamic selector for the page navigation number based on the page for 2 cases:</p>

<ul>
<li>if pagination number is not a link then you are on that page</li>
<li>if pagination number is a link you can use it to click</li>
</ul>

<p>You should't need 4 categories since:
- you can count the number of results and how many can be displayed on a page
- know the number of pages</p>

<ol>
<li>Create a method to navigate if needed with a for or other control structure</li>
<li>For each navigation do what you need to do</li>
</ol>

<p>Or go to the last page and start backwards until page 1 is not a link.</p>

python loops selenium selenium-webdriver pagination