Iterating through a list of links and scraping with Selenium
<p>When I try to iterate a through a list of links and visit them with Selenium with this code:</p>

<pre><code># create link list
urlList = []
with open('my.txt','r') as f: 
for i in f:
    urlList.append(i)


# navigate to URL 
for i in (urlList):
    getUrl = driver.get(i)
    driver.implicitly_wait(3)
</code></pre>

<p>I receive this error:</p>

<blockquote>
  <blockquote>
    <blockquote>
      <p>selenium.common.exceptions.WebDriverException: Message: unknown error: unhandled inspector error: {"code":-32603,"message":"Cannot navigate to invalid URL"}
      (Session info: chrome=51.0.2704.106)
      (Driver info: chromedriver=2.9.248304,platform=Linux 4.2.0-16-generic x86_64</p>
    </blockquote>
  </blockquote>
</blockquote>

<p>Apparently the for loop is generating newline characters from the list and feeding them into the driver.get method. How do I get it to feed urls instead?</p>

<p>If your urls read from the file are getting newlines mixed in, try:</p>

<pre><code>with open('my.txt','r') as f: 
    for i in f:
        urlList.append(i.strip())
</code></pre>

<p>This will remove leading and trailing whitespace from each <code>i</code>. Also, the <code>\n</code>s are not being generated by the loop, they exist in your file which probably has a url on each line, and a <code>'\n'</code> at the end of each line.</p>

python selenium screen-scraping