unable to scrape text
<p>I was trying to get the title of the websites.
So, I used this snippet to do this</p>

<pre><code>    sys.stdout = open("test_data.txt", "w")
    url2 = "https://www.google.com/"
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A'}
    req = urllib2.Request(url2, None, headers)
    req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8')
    html = urllib2.urlopen(req, timeout=60).read()
    soup = BeautifulSoup(html)
    # Extract title
    list1 = soup.title.string
    print list1.encode('utf-8')
</code></pre>

<p>This works perfectly and gives <strong>Google</strong> as the title and flushes the output to test_data.txt.</p>

<p>But when I try to run the same code as a web service, it doesn't work. I get an empty text file.
I am hitting this URL to run this web service on my local <a href="http://0.0.0.0:8881/get_title" rel="nofollow">http://0.0.0.0:8881/get_title</a></p>

<pre><code>from bottle import route, run, request

@route('/get_title')
def get_title():
    sys.stdout = open("test_data.txt", "w")
    url2 = "https://www.google.com/"
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A'}
    req = urllib2.Request(url2, None, headers)
    req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8')
    html = urllib2.urlopen(req, timeout=60).read()
    soup = BeautifulSoup(html)
    # Extract title
    list1 = soup.title.string
    print list1.encode('utf-8')

if __name__ == "__main__":
    run(host='0.0.0.0', port=8881, debug=True)
</code></pre>

<p>Another thing which has made me even more anxious is when I run the web service for msn.com, it works well for both snippets(even the web service).</p>

<p>Any help would be thankful !!</p>

<p>Is this Flask? If so, you need to <code>return</code> the string you want to send to the user. A <code>print</code> statement writes to the web server log. You should replace the last line of your <code>get_title</code> function with this:</p>

<pre><code>return list1.encode('utf-8')
</code></pre>

python web-services scrape