two charset in one website,how to parse
<p>i was study python scraping knowledge recently,and i want to scrap a website</p>

<p><a href="http://news.sina.com.cn/" rel="nofollow">a website with two charset,utf-8 and gb2312</a></p>

<p>i get the warning from beautifulsoup: </p>

<blockquote>
  <p>Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.</p>
</blockquote>

<p>i google the problem and i think it may be the decode problem,my code can scrap other website smoothly.</p>

<p>so ,what should i do?</p>

<p>this is my code:</p>

<pre><code>from urllib.request import urlopen
from bs4 import BeautifulSoup


code_type = 'utf-8'
html = urlopen("http://news.sina.com.cn/")
print(html)

bsObj = BeautifulSoup(html, "html.parser",from_encoding=code_type)

imglist = bsObj.findAll("img")
print(imglist)
</code></pre>

<p>Detect the page coding before parsing it.</p>

<p>Reference to chardet repo
<a href="https://github.com/chardet/chardet" rel="nofollow">https://github.com/chardet/chardet</a>.</p>

<p>You should't set the code_type as utf-8 for all pages, detect the right encoding of page and modify it to the right one.</p>

<p>Encoding detecting might be failed sometime. In that case, you should prepare a dict to store known websites' encoding and use the dict while parsing special pages.</p>

python web-scraping beautifulsoup