Python, bs4: Tags in inspection are nowhere to be found when parsing
<p>I run into an unexpected problem, I am using Python 3.5 and BeautifulSoup.
I want to parse the following link:</p>

<pre><code>url = 'https://www.leboncoin.fr/chaussures/627533472.htm?ca=16_s'
import requests, bs4
res = requests.get(url)
res.raise_for_status()
DicoSoup = bs4.BeautifulSoup(res.text, "lxml")
</code></pre>

<p>I am interested in retrieving the link to the pictures that are in the offer.
When I inspect the html of the website I found that there are to be found under the tag div with class 'thumbnails', they are under tag span with class 'item_imagePic', they are img tags</p>

<p>However, when I select the div tag, the span tags are nowhere to be found:</p>

<pre><code>div = DicoSoup.select("div.thumbnails")

div
Out[54]: 
[&lt;div class="thumbnails" data-alt="Talons aiguilles StÃ©phane KÃ©lian - 37.5"&gt;
 &lt;ul&gt;
 &lt;li class="thumb selected trackable" data-info='{"event_name" :             "ad_view::photos", "event_type" : "click", "click_type" : "N", "event_s2" : "2"}' id="thumb_0"&gt;&lt;/li&gt;
 &lt;li class="thumb trackable" data-info='{"event_name" : "ad_view::photos", "event_type" : "click", "click_type" : "N", "event_s2" : "2"}' id="thumb_1"&gt;          &lt;/li&gt;
 &lt;li class="thumb trackable" data-info='{"event_name" : "ad_view::photos", "event_type" : "click", "click_type" : "N", "event_s2" : "2"}' id="thumb_2"&gt;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;/div&gt;]
</code></pre>

<p>When I inspect the html content, here is what I see:</p>

<pre><code>&lt;div class="thumbnails" data-alt="Talons aiguilles StÃ©phane KÃ©lian - 37.5" style="width: 596px;"&gt;
                        &lt;ul style=""&gt;

                                &lt;li id="thumb_0" class="thumb selected trackable" data-info="{&amp;quot;event_name&amp;quot; : &amp;quot;ad_view::photos&amp;quot;, &amp;quot;event_type&amp;quot; : &amp;quot;click&amp;quot;, &amp;quot;click_type&amp;quot; : &amp;quot;N&amp;quot;, &amp;quot;event_s2&amp;quot; : &amp;quot;2&amp;quot;}"&gt;&lt;span class="item_imagePic"&gt;&lt;img src="//img0.leboncoin.fr/thumbs/d89/d89c778e852e4a175d5d1ba96b2ec9c220445732.jpg" alt="Talons aiguilles StÃ©phane KÃ©lian - 37.5"&gt;&lt;/span&gt;&lt;/li&gt;

                                &lt;li id="thumb_1" class="thumb trackable" data-info="{&amp;quot;event_name&amp;quot; : &amp;quot;ad_view::photos&amp;quot;, &amp;quot;event_type&amp;quot; : &amp;quot;click&amp;quot;, &amp;quot;click_type&amp;quot; : &amp;quot;N&amp;quot;, &amp;quot;event_s2&amp;quot; : &amp;quot;2&amp;quot;}"&gt;&lt;span class="item_imagePic"&gt;&lt;img src="//img1.leboncoin.fr/thumbs/7d9/7d9b62d9efd2187472dc16ca2794be1bbaeb1370.jpg" alt="Talons aiguilles StÃ©phane KÃ©lian - 37.5"&gt;&lt;/span&gt;&lt;/li&gt;

                                &lt;li id="thumb_2" class="thumb trackable" data-info="{&amp;quot;event_name&amp;quot; : &amp;quot;ad_view::photos&amp;quot;, &amp;quot;event_type&amp;quot; : &amp;quot;click&amp;quot;, &amp;quot;click_type&amp;quot; : &amp;quot;N&amp;quot;, &amp;quot;event_s2&amp;quot; : &amp;quot;2&amp;quot;}"&gt;&lt;span class="item_imagePic"&gt;&lt;img src="//img2.leboncoin.fr/thumbs/288/28865002bb34bad516574bd1e9b42d2a2bb928f2.jpg" alt="Talons aiguilles StÃ©phane KÃ©lian - 37.5"&gt;&lt;/span&gt;&lt;/li&gt;

                        &lt;/ul&gt;
                    &lt;/div&gt;
</code></pre>

<p>How is it possible?
What do I need to do to select them?</p>

<p>I have tried:</p>

<pre><code>div = DicoSoup.select_one("div.thumbnails span.item_imagePic")
div = DicoSoup.select_one("div.thumbnails ul li span.item_imagePic")
div = DicoSoup.select("div.thumbnails ul li span.item_imagePic")
span = DicoSoup.find('span', {'class': 'item_imagePic'})
span = DicoSoup.find('span',id="thumb_0")
div = DicoSoup.select("div.thumbnails img")
div = DicoSoup.select("div.thumbnails span img")
div = DicoSoup.select("div.thumbnails ul li span.item_imagePic img")
</code></pre>

<p>They all return me objects of type 'NoneType' </p>

<p>Thanks,</p>

<p>As I commented the thumbnails are dynamically generated using JS but you can get the script and parse the paths:</p>

<pre><code>soup = BeautifulSoup(requests.get("https://www.leboncoin.fr/chaussures/627533472.htm?ca=16_s").content)

script = soup.select_one("div.thumbnails").find_next("script")
print(script.text.strip())
</code></pre>

<p>That gives you:</p>

<pre><code>var images = new Array(), images_thumbs = new Array();
                        images_thumbs[0] = "//img0.leboncoin.fr/thumbs/d89/d89c778e852e4a175d5d1ba96b2ec9c220445732.jpg"; 
              images[0] = "//img0.leboncoin.fr/images/d89/d89c778e852e4a175d5d1ba96b2ec9c220445732.jpg";

                        images_thumbs[1] = "//img1.leboncoin.fr/thumbs/7d9/7d9b62d9efd2187472dc16ca2794be1bbaeb1370.jpg"; 
              images[1] = "//img1.leboncoin.fr/images/7d9/7d9b62d9efd2187472dc16ca2794be1bbaeb1370.jpg";

                        images_thumbs[2] = "//img2.leboncoin.fr/thumbs/288/28865002bb34bad516574bd1e9b42d2a2bb928f2.jpg"; 
              images[2] = "//img2.leboncoin.fr/images/288/28865002bb34bad516574bd1e9b42d2a2bb928f2.jpg";
</code></pre>

<p>To get the images links:</p>

<pre><code>import re


soup = BeautifulSoup(requests.get("https://www.leboncoin.fr/chaussures/627533472.htm?ca=16_s").content)

script = soup.select_one("div.thumbnails").find_next("script").text

print(re.findall("images_thumbs\[\d+\]\s+=\s+\"(.*?)\";", script))
</code></pre>

<p>Or just splitlines and strip:</p>

<pre><code> [s.split("=", 1)[1].strip('"; ') for s in script.splitlines() if s.strip().startswith("images_thumbs")]
</code></pre>

<p>Both give you:</p>

<pre><code>[u'//img0.leboncoin.fr/thumbs/d89/d89c778e852e4a175d5d1ba96b2ec9c220445732.jpg', u'//img1.leboncoin.fr/thumbs/7d9/7d9b62d9efd2187472dc16ca2794be1bbaeb1370.jpg', u'//img2.leboncoin.fr/thumbs/288/28865002bb34bad516574bd1e9b42d2a2bb928f2.jpg']
[u'//img0.leboncoin.fr/thumbs/d89/d89c778e852e4a175d5d1ba96b2ec9c220445732.jpg', u'//img1.leboncoin.fr/thumbs/7d9/7d9b62d9efd2187472dc16ca2794be1bbaeb1370.jpg', u'//img2.leboncoin.fr/thumbs/288/28865002bb34bad516574bd1e9b42d2a2bb928f2.jpg']
</code></pre>

<p>Last all you need is to prepend the scheme which is <em>https</em>:</p>

<pre><code> ["https://"+ path for path in re.findall("images_thumbs\[\d+\]\s+=\s+\"(.*?)\";", script)]
</code></pre>

python parsing tags beautifulsoup