Encoding error when reading url with urllib
<p>When I try to scrape a wikipedia site with a special character in its URL, using urllib.request and Python, I get the following error <code>UnicodeEncodeError: 'ascii' codec can't encode character '\xf8' in position 23: ordinal not in range(128)</code></p>

<p>The code:</p>

<pre><code># -*- coding: utf-8 -*-
import urllib.request as ur

url = "https://no.wikipedia.org/wiki/Jonas_Gahr_StÃ¸re"
r = ur.urlopen(url).read()
</code></pre>

<p>How can I use urllib.request with utf-8 encoding?</p>

<p>Apparently, urllib can only handle ASCII requests, and converting your url to ascii gives a error on your special character.
Replacing Ã¸ with %C3%B8, the proper way to encode this special character in http, seems to do the trick. However, I can't find a method to do this automatically like your browser does.</p>

<p>example:</p>

<pre><code>&gt;&gt;&gt; f="https://no.wikipedia.org/wiki/Jonas_Gahr_St%C3%B8re"
&gt;&gt;&gt; import urllib.request
&gt;&gt;&gt; g=urllib.request.urlopen(f)
&gt;&gt;&gt; text=g.read()
&gt;&gt;&gt; text[:100]
b'&lt;!DOCTYPE html&gt;\n&lt;html class="client-nojs" lang="nb" dir="ltr"&gt;\n&lt;head&gt;\n&lt;meta charset="UTF-8"/&gt;\n&lt;title'
</code></pre>

<p>The answer above doesn't work, because he is encoding after the request is processed, while you get an error during the request processing.</p>

python urllib