Using Selenium to extract data provided by an autocomplete search
<p>I want to extract part of the result provided by a site's search bar's auto complete. I'm having trouble extracting the result. I'm able to enter the query I want, but I'm unable to store the autosuggestion. It seems whenever I click the drop down suggestions to "inspect element" in order to find what to select the drop down menu vanishes! </p>

<p>Here's the code I'm working with:</p>

<pre><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import os
from scrapy.selector import HtmlXPathSelector

#launch chromedirver
driver.get("http://www.marinetraffic.com/en/ais/index/ports/all")

searchBox = WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located(
        (By.XPATH, '//input[@id= "portname"]')
    )
)
searchBox.click()
searchBox.clear()
a = searchBox.send_keys('Belawan') #so far so good

selen_html = driver.find_element_by_class_name('input-group').get_attribute('innerHTML')
hxs = HtmlXPathSelector(text=selen_html)
suggests =  hxs.select('//div[@class= "input-group"/Belawan/@title').extract
driver.close()
</code></pre>

<p>The error, unsurprisingly, is <code>ValueError: XPath error: Invalid predicate in //div[@....[etc]</code>. How do I find the correct name to put in my XPath? </p>

<p>The autocomplete takes the form <code>BELAWAN - Port [ID]</code> the end goal is to pull out <code>ID</code>. </p>

<p>Edit: 
<a href="http://imgur.com/a/cjpXk" rel="nofollow">screenshot</a></p>

<p>This should work.
Basically you will have find the xpath locators of those web elements'</p>

<p>In your case it was like </p>

<pre><code>&lt;ul class="ui-autocomplete ui-front ui-menu ui-widget ui-widget-content ui-corner-all" id="ui-id-3" tabindex="0" style="display: none; top: 375px; left: 63px; width: 306px;"&gt;
   &lt;li class="ui-menu-item" role="presentation"&gt;&lt;a id="ui-id-7" class="ui-corner-all" tabindex="-1"&gt;&lt;b&gt;BELA&lt;/b&gt;WAN&amp;nbsp;-&amp;nbsp;Port [ID]&lt;/a&gt;&lt;/li&gt;
   &lt;li class="ui-menu-item" role="presentation"&gt;&lt;a id="ui-id-8" class="ui-corner-all" tabindex="-1"&gt;&lt;b&gt;BELA&lt;/b&gt;WAN ANCH&amp;nbsp;-&amp;nbsp;Ancorage [ID]&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</code></pre>

<p>So I used id to get the other <code>ul</code> and then used <code>find_elements_by_xpath</code> to get list of childrend matching the xpath.</p>

<pre><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import os

#launch chromedirver
driver = webdriver.Chrome()
driver.get("http://www.marinetraffic.com/en/ais/index/ports/all")

searchBox = WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located(
        (By.XPATH, '//input[@id= "portname"]')
    )
)
searchBox.click()
searchBox.clear()
a = searchBox.send_keys('Belawan') #so far so good

web_elem_list = driver.find_element_by_id("ui-id-3").find_elements_by_xpath("//li[@role='presentation']/a")
suggests = [web_elem.text for web_elem in web_elem_list]
driver.close()
print suggests


# Will Give o/p
[u'BELAWAN - Port [ID]', u'BELAWAN ANCH - Ancorage [ID]']
</code></pre>

python selenium selenium-chromedriver