Iterating through a list of links and scraping with Selenium
<p>When I try to iterate a through a list of links and visit them with Selenium with this code:</p>

<pre><code># create link list
urlList = []
with open('my.txt','r') as f: 
for i in f:
    urlList.append(i)


# navigate to URL 
for i in (urlList):
    getUrl = driver.get(i)
    driver.implicitly_wait(3)
</code></pre>

<p>I receive this error:</p>

<blockquote>
  <blockquote>
    <blockquote>
      <p>selenium.common.exceptions.WebDriverException: Message: unknown error: unhandled inspector error: {"code":-32603,"message":"Cannot navigate to invalid URL"}
      (Session info: chrome=51.0.2704.106)
      (Driver info: chromedriver=2.9.248304,platform=Linux 4.2.0-16-generic x86_64</p>
    </blockquote>
  </blockquote>
</blockquote>

<p>Apparently the for loop is generating newline characters from the list and feeding them into the driver.get method. How do I get it to feed urls instead?</p>

<p>I run your program on my computer, But I don not get any error?</p>

<p>This is <code>my.txt</code> file, there are two china website url in it:</p>

<pre><code>https://www.baidu.com/
https://www.sogou.com/
</code></pre>

<p>This is <code>test.py</code> file, It will <code>get</code> the websites in <code>my.txt</code>:</p>

<pre><code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-


import time
from selenium import webdriver

driver = webdriver.Chrome()  # Optional argument, if not specified will search path.
urlList = []
with open('my.txt', 'r') as f:
    for i in f:
        urlList.append(i)


for i in (urlList):
    print(i)
    getUrl = driver.get(i)
    time.sleep(3)
    driver.implicitly_wait(3)
</code></pre>

<p>These are the output of my program:</p>

<pre><code>â /tmp/selenium $ python3 test.py
https://www.baidu.com/

https://www.sogou.com/
</code></pre>

<p>So I think there may be other mistakes in your program. Can you show the content of the <code>my.txt</code> and the complete code?</p>

python selenium screen-scraping