Python Requests declined: "Due to the presence of characters known to be used in Cross Site Scripting attacks, access is forbidden."
<p>Dear fellow <code>requests</code> users,</p>

<p><strong>Update:</strong></p>

<p>Sorry, guys. My error came from a mistake:</p>

<p>My goal was to do this:</p>

<pre><code>r = requests.get('http://www.spdrs.com/product/fund.seam?ticker=SPY', stream=True, headers=hdr)
</code></pre>

<p>I did this: </p>

<pre><code>r = requests.get('http://www.spdrs.com/product/fund.seam?ticker={}'.format(['SPY']), stream=True, headers=hdr)
</code></pre>

<p>Which should be:</p>

<pre><code>r = requests.get('http://www.spdrs.com/product/fund.seam?ticker={}'.format('SPY'), stream=True, headers=hdr)
</code></pre>

<p>The extra brackets <strong><code>[]</code></strong> were causing the error, apparently. Dumb mistake. Feel free to vote me down, if you wish.</p>

<p><strong>Original question:</strong></p>

<p>I am trying to scrape <code>spdrs.com</code> webpage using:</p>

<pre><code>hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
   'Accept-Encoding': 'none',
   'Accept-Language': 'en-US,en;q=0.8',
   'Connection': 'keep-alive'}    
r = requests.get('http://www.spdrs.com/product/fund.seam?ticker=SPY', stream=True, headers=hdr)
</code></pre>

<p>But all I get is this:</p>

<pre><code>Due to the presence of characters known to be used in Cross Site Scripting attacks, access is forbidden.
</code></pre>

<p>It's the same with <code>http</code> or <code>https</code>.</p>

<p>If I remove <code>hdr</code>, I get a straight 403 decline.</p>

<p>Is there any modification I can do to the <code>hdr</code> to show the website that I am a well-behaving script? I know, servers don't like scrapers.</p>

<p>This <a href="http://stackoverflow.com/questions/5249130/due-to-the-presence-of-characters-known-to-be-used-in-cross-site-scripting-attac">thread on SO</a> shows a similar problem from webmaster's perspective.</p>

<p>Thanks a lot!</p>

<p>Yi</p>

<p>Drop the following key/value from your header:</p>

<pre><code>'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
</code></pre>

<p>It works for me after I did that.</p>

python python-requests