Scrapy extracting the wrong IMG SRC
<p>I'm trying to use Scrapy to get the <a href="https://www.tripadvisor.com/Attraction_Review-g155032-d1494256-Reviews-Gray_Line-Montreal_Quebec.html" rel="nofollow">URLs of images on a page</a> with ID <code>HERO_PHOTO</code>. The target element has the following HTML code</p>

<p><code>&lt;img alt="Photo of Gray Line" style="position: relative; left: -50px; top: 0px;" id="HERO_PHOTO" class="flexibleImage" src="https://media-cdn.tripadvisor.com/media/photo-s/04/71/70/7c/gray-line-tours-montreal.jpg" width="352" height="260"&gt;</code></p>

<p>Within Chrome browser, running</p>

<pre><code>$('#HERO_PHOTO').attr('src')
</code></pre>

<p>grabs the URL correctly</p>

<pre><code>"https://media-cdn.tripadvisor.com/media/photo-s/04/71/70/7c/gray-line-tours-montreal.jpg"
</code></pre>

<hr>

<p><strong>Problem</strong>:
However using the following CSS selector in Scrapy,</p>

<pre><code>response.css('#HERO_PHOTO::attr(src)').extract_first()
</code></pre>

<p>and</p>

<pre><code>response.css('#HERO_PHOTO').xpath('@src').extract_first()
</code></pre>

<p>and</p>

<pre><code>response.css('#HERO_PHOTO[src]').extract_first()
</code></pre>

<p>is giving us</p>

<pre><code>https://static.tacdn.com/img2/x.gif
</code></pre>

<p>Using <code>.extract()</code> also returned the same incorrect URL.</p>

<p>Why is Scrapy grabbing a different SRC value?</p>

<p>I believe you are using the wrong css selector. Looking at <a href="http://www.w3schools.com/cssref/css_selectors.asp" rel="nofollow">w3 schools</a> it seems to select your attribute you want [src].</p>

<p>Try this.</p>

<p>response.css('#HERO_PHOTO[src]').extract_first()</p>

<p>my next suggestion is to see what you get without using the extract_first(). See if it's in the return value of response.css('#HERO_PHOTO[src]')</p>

<p><strong>EDIT:</strong> I think the issue you're experiencing is you are querying the page source, not the rendered html. Here's a link to what I believe is happening.</p>

<p><a href="http://stackoverflow.com/questions/18244908/inspect-element-and-view-source-code-are-showing-two-different-things">This Questions first answer</a></p>

<p>You are querying what the server had responded, not what JavaScript has had a chance to manipulate.</p>

python css python-2.7 scrapy