Untraceable HTTP redirection?
<p>I'm currently working on a project to track products from several websites. I use a python scraper to retrieve all the URLs related to the listed products, and later, regularly check if these URLs are still active.</p>

<p>To do so I use the Python requests module, run a get request and look at the response's status code. Usually I get <strong>200</strong>, <strong>301</strong>, <strong>302</strong> or <strong>404</strong> as expected, except in the following case:</p>

<p><a href="http://www.sephora.fr/Parfum/Parfum-Femme/Totem-Orange-Eau-de-Toilette/P2232006" rel="nofollow">http://www.sephora.fr/Parfum/Parfum-Femme/Totem-Orange-Eau-de-Toilette/P2232006</a></p>

<p>This product has been removed and while opening the link (sorry it's in French), I am briefly shown a placeholder page saying the product is not available anymore and then redirected to the home page (www.sephora.fr).</p>

<p>Oddly, Python still returns a <strong>200</strong> status code and so do various redirect tracers such as wheregoes.com or redirectdetective.com. The worst part is that the response URL still is the original, so I can't even trace it that way.</p>

<p>When analyzing with Chrome DevTools and preserving the logs, I see that at some point the page is reloaded. However I'm unable to find out where. </p>

<p>I'm guessing this is done client-side via Javascript, but I'm not quite sure how. Furthermore, I'd really need to be able to detect this change from within Python.</p>

<p>As a reference, here's a link to a working product:</p>

<p><a href="http://www.sephora.fr/Parfum/Parfum-Femme/Kenzo-Jeu-d-Amour-Eau-de-Parfum/P1894014" rel="nofollow">http://www.sephora.fr/Parfum/Parfum-Femme/Kenzo-Jeu-d-Amour-Eau-de-Parfum/P1894014</a></p>

<p>Any leads?</p>

<p>Thank you !
Ludwig</p>

<p>The page has a <a href="https://en.wikipedia.org/wiki/Meta_refresh" rel="nofollow">meta tag</a>, that redirects the page to the root URL:</p>

<pre class="lang-html prettyprint-override"><code>&lt;meta http-equiv="refresh" content="0; URL=/" /&gt;
</code></pre>

javascript python http redirect