Web scraping asian language sites with Python
<p>Not very familiar with the Python ecosystem, or with web scraping generally. So I'm trying to scrape content from a Chinese language site. </p>

<pre><code>from bs4 import BeautifulSoup
import requests

r = requests.get("https://www.baidu.com/")
r.encoding = 'utf-8'

text = r.text

soup = BeautifulSoup(text.encode('utf-8','ignore'), 'html.parser')

print soup.prettify()
</code></pre>

<p>The problem is, this code works for me, but it doesn't work for everyone, and I don't know enough about character encoding or the python ecosystem to troubleshoot the issue. I'm running Python 2.7.10, but running this same block of code on another computer with Python 2.7.12 resulted in the following error: "UnicodeEncodeError: 'ascii' codec can't encode chracters in position 369-377: ordinal not in range(128)"</p>

<p>So I guess my question really is the following: </p>

<p>What is causing this error? And how can I fix this code to make it more portable? </p>

<p>Thank you in advance for any guidance or pointers. </p>

<p>I think you don't need to specify the encoding for request.
since the r.text has do the coding-transform work and r.content is the raw data.</p>

<p>see the documents:</p>

<pre><code> |  text
 |      Content of the response, in unicode.
 |      
 |      If Response.encoding is None, encoding will be guessed using
 |      ``chardet``.
 |      
 |      The encoding of the response content is determined based solely on HTTP
 |      headers, following RFC 2616 to the letter. If you can take advantage of
 |      non-HTTP knowledge to make a better guess at the encoding, you should
 |      set ``r.encoding`` appropriately before accessing this property.
</code></pre>

<p>so you just need config the response's encoding, not the requests' encoding.</p>

<p>so the code should be like this:</p>

<pre><code>print r.encoding
r.encoding = "urf8"
print r.text
</code></pre>

python web-scraping beautifulsoup