How to access grand childs of a node using XPath in Python Selenium?
<p>I am preparing a web-scrapping script which supposed to find list of attorneys in an area through a business directory web-site. I use chrome driver to fill out search keywords and the area values. </p>

<p>Since some of the hits don't have phone number, I would like to iterate through list of DIVs corresponding to search results and then check if the it has the phone number as a grand grand child and if yes then I get the phone number too otherwise I leave that field blank. </p>

<p>I have come up with two ways to do it as per below code. </p>

<pre><code>import time
import json as js
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

url = 'http://www.yellowpages.com/search?search_terms=Divorce+Attorneys&amp;geo_location_terms=sun+diego'
RsultsList = []

driver = webdriver.Chrome()
driver.get(url)
ThereIsNext = True
while ThereIsNext:

    # find ads.
    nAddResults = len( driver.find_elements_by_xpath("//div[@class='result flash-ad']"))

    #print 'add size = %d' % nAddResults
    for i in range(nAddResults):
        phone1 = driver.find_elements_by_xpath("//div[@class='result flash-ad']/div[1]/div[1]/div[2]/div[1]/ul[1]/li[1]")[i].text
        BusinessName1 = driver.find_elements_by_xpath("//div[@class='result flash-ad']//a[@class='business-name']")[i].text

        elem = driver.find_elements_by_xpath("//div[@class='result flash-ad']")
        phone2 = elem.find_element_by_xpath("/div[1]/div[1]/div[2]/div[1]/ul[1]/li[1]")[i].text
        BusinessName2 = elem.find_element_by_xpath("//a[@class='business-name']")
</code></pre>

<p>The first one is prone to error as records with no phone no. do NOT necessary show up at the end. So I come up with the second way. However, I get the below error message if try the second.</p>

<pre><code>Traceback (most recent call last):
  File "C:\Users\XXXX\documents\visual studio 2015\Projects\PythonApplication3\
PythonApplication3\AtorneyList.py", line 23, in &lt;module&gt;
    phone2 = elem.find_element_by_xpath("/div[1]/div[1]/div[2]/div[1]/ul[1]/li[1
]").text
AttributeError: 'list' object has no attribute 'find_element_by_xpath'
Press any key to continue . . .
</code></pre>

<p>Please let me know what I am missing. I have checked  <a href="http://stackoverflow.com/questions/9548523/how-do-i-find-node-and-its-child-nodes-using-selenium-with-python?s=1%7C1.1269">this</a> and <a href="http://stackoverflow.com/questions/9548523/how-do-i-find-node-and-its-child-nodes-using-selenium-with-python/9548919?s=2%7C0.1580#9548919">that</a> already and couldn't understand.</p>

<p>Much appreciate it.</p>

<p>Thanks</p>

<p>Actually <code>find_elements()</code> returns either list of <a href="http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement" rel="nofollow"><code>WebElement</code></a> or empty list. You're storing this result into a list variable name <code>elem</code>.</p>

<blockquote>
  <p>AttributeError: 'list' object has no attribute 'find_element_by_xpath'</p>
</blockquote>

<p>This occurs because you're going to find nested <a href="http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement" rel="nofollow"><code>WebElement</code></a> on <code>elem</code> list that's why <strong>you're calling as <code>elem.find_element_by_xpath()</code> which is absolutely wrong.</strong></p>

<p>Actually <a href="http://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webelement.WebElement.find_element" rel="nofollow"><code>find_element()</code></a> or <a href="http://selenium-python.readthedocs.io/api.html#selenium.webdriver.remote.webelement.WebElement.find_elements" rel="nofollow"><code>find_elements</code></a> is used to search the element on the <strong>page context</strong> or the <strong>context of the <a href="http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement" rel="nofollow"><code>WebElement</code></a></strong> instead of <code>list</code>. </p>

<p>So you should try to find list of <a href="http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement" rel="nofollow"><code>WebElement</code></a> from <code>driver</code> means page context and then iterate to find further nested <a href="http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement" rel="nofollow"><code>WebElement</code></a> using this element context as below :-</p>

<pre><code>elems = driver.find_elements_by_xpath("//div[@class='result flash-ad']")

for elem in elems:
    phone2 = elem.find_element_by_xpath(".//div[1]/div[1]/div[2]/div[1]/ul[1]/li[1]").text
    BusinessName2 = elem.find_element_by_xpath(".//a[@class='business-name']").text
</code></pre>

python selenium xpath web-scraping