Making Python go a webpage that has a lot of links, scraping each link and seeing if they have a specific piece of text
<p><strong>I have a question pertaining to Python that I hope can be remedied.</strong>
I'm not asking to be spoonfed, but any advice will be extremely helpful. </p>

<p>I'm working on a mini-project of sorts where I "crawl" the WW1 Database of Canadian Soldiers who died and seeing which pages lack info. </p>

<p><a href="http://www.canadaatwar.ca/memorial/world-war-i/" rel="nofollow">http://www.canadaatwar.ca/memorial/world-war-i/</a></p>

<p>I'm trying to make Python go to each soldiers page, and seeing if the "biography" section is empty. </p>

<p>This is my code (not mine, I'll give credit and link the original page later) so far. It's very messy, and it may make senior developers tear their hair out in frustration, but bear with me.</p>

<pre><code>import requests
from bs4 import BeautifulSoup
import time
url_to_scrape = 'http://www.canadaatwar.ca/memorial/world-war-i/'
r = requests.get(url_to_scrape)
soup = BeautifulSoup(r.text)
soldier_links = []
for table_row in soup.select(".soldierslist tr"):
table_cells = table_row.findAll('td')
if len(table_cells) &gt; 0:
relative_link_to_soldier_details = table_cells[0].find('a')['href']
absolute_link_to_soldier_details = url_to_scrape +   relative_link_to_soldier_details
soldier_links.append(absolute_link_to_soldier_details)
inmates = []
r = requests.get(soldier_link)
soup = BeautifulSoup(r.text)
soldier_details = {}
soldier_profile_rows = soup.select("#soldierProfile tr")
soldier_details['additional text information is avalable on this    individual just yet. If you have more information please'] =    soldier_profile_rows[0].findAll('td')[0].text.strip()
soldiers.append(soldier_details)
</code></pre>

<p>It would be great to know how to make Python go to the next page once it's done scraping everything from the page it's scraping, and making Python print only the links that have information in the biography section. </p>

<p>I would like suggest you use Scrapy Framework. In building scrapy spiders, </p>

<ol>
<li>You can specify how to start request, e.g., the simplest way by setting <code>start_urls</code> or overwrite function <code>start_request</code></li>
<li>You can specify how to parse response. Here you can do the xpath selection, if the condition is satisfied, yield the result as items and at same time yield a new request (in this way, you are following the next request).</li>
</ol>

<p>Hope this would be helpful.</p>

python