UnicodeEncodeError Only When Script is Run as a Subprocess
<p>I'm running my main script in Python 3.5 using the Spyder IDE, and I want to import functions from a script that happens to only work in Python 3.4. So I was recommended to run this second script as a subprocess like so:</p>

<pre><code>import subprocess
cmd = [r'c:\python34\pythonw.exe', r'C:\users\John\Desktop\scraper.py']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
print(stdout)
print(stderr)
</code></pre>

<p>The script being called is an example from NikolaiT's Web engine scraper:</p>

<pre><code># -*- coding: utf-8 -*-

import sys
from GoogleScraper import scrape_with_config, GoogleSearchError
from GoogleScraper.database import ScraperSearch, SERP, Link

def basic_usage():
    # See in the config.cfg file for possible values
    config = {
        'SCRAPING': {
            'use_own_ip': 'True',
            'search_engines': 'baidu',
            'num_pages_for_keyword': 3
        },
        'keyword': 'è¹æ',
        'SELENIUM': {
            'sel_browser': 'chrome',
        },
        'GLOBAL': {
            'do_caching': 'False'
        }
    }

    try:
        sqlalchemy_session = scrape_with_config(config)
    except GoogleSearchError as e:
        print(e)

    # let's inspect what we got
    link_list = []
    for serp in sqlalchemy_session.serps:
        #print(serp)
        for link in serp.links:
            #print(link)
            link_list.append(link.link)
    return link_list



links = basic_usage()

print("test")
for link in links:
    print(link)
</code></pre>

<p>This script works just fine when running it in Python 3.4's IDLE IDE, but when running it as a subprocess as above, I get the following UnicodeEncodeError printed from my main script:</p>

<p>\python34\scripts\googlescraper\GoogleScraper\caching.py", line 413,
in parse_all_cached_files store_serp_result(serp, self.config) </p>

<p>File "c:\python34\scripts\googlescraper\GoogleScraper\output_converter.py", line 123, in store_serp_result pprint.pprint(data) </p>

<p>File "c:\python34\lib\pprint.py", line 52, in pprint    printer.pprint(object) </p>

<p>File "c:\python34\lib\pprint.py", line 139, in pprint self._format(object, self._stream, 0, 0, {}, 0) </p>

<p>File "c:\python34\lib\pprint.py", line 193, in _format allowance + 1, context, level) </p>

<p>File "c:\python34\lib\pprint.py", line 268, in _format write(rep) </p>

<p>File "c:\python34\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] 
UnicodeEncodeError: \'charmap\' codec can\'t encode characters in position 1-2: character maps to undefined'</p>

<p>Why would this only happen when running it indirectly? Thanks for any help, clarification questions, or suggestions for improvement in framing my question.</p>

<p>In short, your issue is as follows:</p>

<ul>
<li>Python IDE fully supports UTF-8 encoding, and that's why your invoked script runs fine there.</li>
<li>On the other hand, when opening a subprocess, since you are on Windows, you are using Windows1252 charset by default, which doesn't support part of your output.</li>
</ul>

<p>Quick solution: If you invoke the script as <code>python foobar.py</code> on Windows command line, you can run <code>chcp 65001</code> before invoking the script; if you use Spyder IDE, there should be a setting that allows you to set file / project encoding to UTF-8.</p>

<p>(You may also try adding <code># -*- coding: utf-8 -*-</code> to the top of your main Python script.)</p>

python subprocess