Selenium scrolling internal scroll bar and scrapping results
<p>I'm trying to scrape <a href="http://comparefirst.sg/wap/productsListEvent.action?prodGroup=whole&amp;pageAction=prodlisting" rel="nofollow">this website</a> for my project to populate a list of insurance products available.</p>

<p>However, the website has an internal scrolling bar, that only displays the first 10 items onto the page, and would only bring new elements onto display when you scroll that internal bar downwards.</p>

<p>How do I </p>

<ul>
<li>Use python <code>Selenium</code> to scroll that internal bar downwards? Can't seem to find much information of that around.</li>
<li>How do I use <code>Selenium</code> to retrieve the <code>Company Name, Product Name, Paymode, product features (if active)</code> and return a <code>pandas Dataframe</code>?</li>
</ul>

<p>Interesting thing is, <em>you don't need to scroll the container at all.</em> All the results are actually loaded, but part of them are just invisible. You can simply find all <code>li</code> elements with <code>result_content</code> class and get the desired data.</p>

<p>Example working code extracting the "prod names":</p>

<pre><code>from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver


driver = webdriver.Chrome("/usr/local/bin/chromedriver")
driver.maximize_window()
driver.get("http://comparefirst.sg/wap/productsListEvent.action?prodGroup=whole&amp;pageAction=prodlisting")

wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.ID, "result_container")))
results = driver.find_elements_by_css_selector("li.result_content")

for result in results:
    prod_name = result.find_element_by_id('sProdName').get_attribute("innerText")
    print(prod_name)

driver.close()
</code></pre>

<p>Prints:</p>

<pre><code>AIA Gen3 (II)
AIA Guaranteed Protect Plus
AIA Guaranteed Protect Plus
...
DIRECT- TM Basic Whole Life
DIRECT- TM Basic Whole Life (+ Critical Illness)
TM Legacy
TM Legacy (+ Critical Illness)
TM Legacy LifeFlex
TM Legacy LifeFlex (+ Critical Illness)
TM Retirement GIO
TM Retirement PaycheckLife (Single Life)
</code></pre>

<p>Note that we have to use <code>.get_attribute("innerText")</code> instead of <code>.text</code> since the latter would return the visible text only while most of our elements are invisible.</p>

python selenium