Unable to scrape the content though it appears in the inspect element
<p>I have been trying to scrape the user reviews from the cnet page. The pros and cons of the user reviews information. (<a href="http://www.cnet.com/products/samsung-galaxy-s7/user-reviews/" rel="nofollow">http://www.cnet.com/products/samsung-galaxy-s7/user-reviews/</a>)</p>

<p>I have used selenium to dynamically load the page but still the html source and inspect element source are different. I also have used requests to get the source code. I am not sure about the difference in between the two.</p>

<p>Can you please suggest me a work around? </p>

<p>The code used for selenium:</p>

<pre><code>driver.get("http://www.cnet.com/products/samsung-galaxy-s7/user-reviews/")
driver.wait = WebDriverWait(driver, 2)
soup= BeautifulSoup(driver.page_source,"html.parser")
</code></pre>

<p>requests code:</p>

<pre><code>try:
    r = requests.get("http://www.cnet.com/products/samsung-galaxy-s7/user-reviews/", timeout = 10)
except Exception,e:
    print("borken")
data = r.text
soup = BeautifulSoup(data)
</code></pre>

<p>PS: I gave a search in stack overflow and google, but I couldn't find a working answer. It would also be helpful if anyone can give me a link. </p>

<p>That page has quite a bit of JavaScript so I think using selenium is going to be your best bet to load all of the content. In your current code you are only waiting 2 seconds which may not be enough time. I would recommend using an explicit wait that returns when the comment section elements have completely loaded.</p>

<p>There is a good explanation and example here: <a href="http://selenium-python.readthedocs.io/waits.html" rel="nofollow">http://selenium-python.readthedocs.io/waits.html</a> </p>

<p>It looks like each of the comment blocks have the form</p>

<pre><code>&lt;article class= "fyre-comment-article fyre-comment-source-0"..../&gt;
</code></pre>

<p>so you could start by modifying the example from the above link like this: (I don't have the ability to actually run this code on my current machine so just use it as an example of a way to modify the code from the link. Read here for more ways to locate elements: <a href="http://selenium-python.readthedocs.io/locating-elements.html" rel="nofollow">http://selenium-python.readthedocs.io/locating-elements.html</a>)</p>

<pre><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = &lt;yourdriver&gt;
driver.get("http://www.cnet.com/products/samsung-galaxy-s7/user-reviews/")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.XPATH,
        '//article[class="fyre-comment-article fyre-comment-source-0"]'))
    )
finally:
    driver.quit()
</code></pre>

python selenium web-scraping