Python Scraping - Unable to get required data from Flipkart
<p>I was trying to scrape the customer reviews from Flipkart website. The following is the <a href="https://www.flipkart.com/samsung-galaxy-j5-6-new-2016-edition-white-16-gb/product-reviews/itmegmrnzqjcpfg9?pid=MOBEG4XWJG7F9A6Z" rel="nofollow">link</a>. The following was my code to scrape, but it is always returning an empty list.</p>

<pre><code>&gt;&gt;&gt; from bs4 import BeautifulSoup
&gt;&gt;&gt; import requests

&gt;&gt;&gt; r = requests.get('https://www.flipkart.com/samsung-galaxy-j5-6-new-2016-edition-white-16-gb/product-reviews/itmegmrnzqjcpfg9?pid=MOBEG4XWJG7F9A6Z')
&gt;&gt;&gt; soup = BeautifulSoup(r.content, 'lxml') # Tried with 'html.parser' also
&gt;&gt;&gt; soup.find_all('div', '_3DCdKt')
[]
&gt;&gt;&gt; soup.find_all('div', {'class': '_3DCdKt'})
[]
&gt;&gt;&gt; soup.find_all('div', {'class': 'row _3wYu6I _3BRC7L'})
[]
&gt;&gt;&gt; soup.find_all('div', {'class': '_1GRhLX hFPo14'})
[]
</code></pre>

<p>So, I tried to get the entire section, but I was getting only the following:</p>

<pre><code>&gt;&gt;&gt; soup.find_all('div', {'class': 'col-9-12'})
[&lt;div class="col-9-12" data-reactid="96"&gt;&lt;div class="row _2_xtR5" data-reactid="97"&gt;&lt;/div&gt;&lt;div class="row _3wYu6I _1KVtzT" data-reactid="98"&gt;&lt;/div&gt;&lt;/div&gt;]
</code></pre>

<p>I was not getting the other contents. So, next I tried with selenium, even then it was returning <code>None</code>. The following is my selenium code:</p>

<pre><code>&gt;&gt;&gt; driver = webdriver.Firefox()
&gt;&gt;&gt; driver.get('https://www.flipkart.com/samsung-galaxy-j5-6-new-2016-edition-white-16-gb/product-reviews/itmegmrnzqjcpfg9?pid=MOBEG4XWJG7F9A6Z')
&gt;&gt;&gt; a = driver.find_elements_by_class_name("_3DCdKt")
&gt;&gt;&gt; len(a)
10
&gt;&gt;&gt; for i in a:
...    print i.get_attribute('value')
...
None
None
None
None
None
None
None
None
None
None
</code></pre>

<p>What might be the problem? Am I doing any mistakes in the code. Kindly help. I am new to Python.</p>

<p>This seemed to work for me</p>

<pre><code>from selenium import webdriver
from bs4 import BeautifulSoup
driver = webdriver.Firefox()
driver.get('https://www.flipkart.com/samsung-galaxy-j5-6-new-2016-edition-white-16-gb/product-reviews/itmegmrnzqjcpfg9?pid=MOBEG4XWJG7F9A6Z')
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
for entry in soup.find_all('div', '_3DCdKt'):
    print(entry.text)
</code></pre>

<p>Perhaps change <code>entry.text</code> to just <code>entry</code>, depending on your needs.</p>

<p>Output was:</p>

<pre><code>5âBest Smartphone by SamsungThanks to Flipkart who deliver it me with in 5 days Good Phone With Metal Body And Best front Camera With FlashBest for night Selfie I Take more than 30 pic in night mode with front flash good smartphone  gold color is also superebbest ever smartphone under 15k by samsungGood BatteryGood Camera Front with Flash and Rear Also SuperbREAD MOREHappy ThakurCertified Buyer16 May, 201621776PermalinkReport Abuse
5âMust BuyThis Phone is awesome..Must BuyREAD MOREDurvank AregekarCertified Buyer10 Aug, 201680PermalinkReport Abuse
3âGood looking phone with some drawbacksHi,I got this phone from Flipkart on Friday and here is my 3 days review.Pros: * Beautiful design * Very handy, easy to handle * Battery backup is great * Back camera is good * No heating issues Cons: * If we are charging, it will not show any light or any notification whether it is charging or not. We need to on the screen and check whether it is charging or not. So every time we need to turn it on and see whether it is charging or not.* Camera issue: Once you take the picture a...READ MOREileep Certified Buyer16 May, 201615167PermalinkReport Abuse
5âAbsolute Stunner and ImpressiveUpdated Review on 02-August after 3 months of usage:What I liked most:Look : 100/100 - Very good looking phone. Gold color and the finishing is super coolSize : 100/100 - 5.2 Inch is neither big nor small. I can still operate with one hand.. Battery : 100/100 - 3100 mAH is outstanding. 3G is always ON when i am out of home and Wi-Fi is always ON in home. I am charging mobile only once in every 36 hours. I use Whatsapp, instagram and Browsing mostly. Display : 90/100 - Not so bright and s...READ MORENaresh KaretiCertified Buyer13 May, 201618787PermalinkReport Abuse
5âIt's very goodNice.battery backup it's goodREAD MOREFlipkart CustomerCertified Buyer17 Aug, 201660PermalinkReport Abuse
5âGood phoneIt is a good phoneREAD MORESourabh JainCertified Buyer9 Aug, 201660PermalinkReport Abuse
5âBrilliant Phone Compared to MoneySuper Amoled Display..2 GB RAM with Latest Android Marshmallow OS only for 13K....its difficult to get Samsung Phone with 2 GB ram in such a low price Range...used for 15 days....Going Smooth....Awesome Earphone Quality.....selfie and back Camera Good.....Battery last for more than a day with Continous usage or will go for two days....Free Microsoft apps and Much More...READ MOREPrashant DiasCertified Buyer7 Sep, 2016269PermalinkReport Abuse
5âvery gooddelivery is in time but my phone is heat will data is on plz checkREAD MORESanthoaha m n santhuCertified Buyer12 Aug, 201681PermalinkReport Abuse
4âBy Expert     -Vaishnav VJGood Product by SamsungThe things from this phone is       1. Marshmellow v6.0       2. Front flash with 5mb camera not so good       3. Its design       4. Primary Camera is not so good with 13mb led flash       5. Battery life is also not so good       6. Its size is correct in its design       7. Supports OTG       9. Only 2GB RAM      10. 16GB Internal storage but only 11GB is availiable      11. 4G supports      12. Ultra power saving mode      13. S bike mode      14. Spe...READ MOREVaishnav Certified Buyer17 May, 2016155PermalinkReport Abuse
4âLook is great but camera is a bit slow.I have been using this phone for the past one month. Look is great and also has metal body. So far I have only noticed a bit slowness in the camera while capturing a photo and switching between your photo gallery and camera. Other than that, phone works fine (good battery back-up) - need to see how long it lastsREAD MOREFlipkart CustomerCertified Buyer17 Aug, 201692PermalinkReport Abuse
</code></pre>

<p>Selenium is slow, so if you don't need it (see Padraic Cunningham answer), I wouldn't use it.</p>

python selenium beautifulsoup