Tornado: How to get and return large data with less memory usage?
<p>I have web-crawler and http interface for it.</p>

<p>Crawler gets grouped urls as dictionary. I need to return a result in the same format in JSON. But I was faced with a large memory usage, which is not returned to the operating system. How can I implement this solution without large memory usage?</p>

<p>Code:
</p>

<pre><code>#!/usr/bin/env python
# coding=utf-8

import collections

import tornado.web
import tornado.ioloop
import tornado.queues
import tornado.httpclient


class ResponseError(Exception):

    pass


class Crawler(object):

    client = tornado.httpclient.AsyncHTTPClient()

    def __init__(self, groups, concurrency=10, retries=3, validators=None):
        self.groups = groups
        self.concurrency = concurrency
        self.retries = retries
        self.validators = validators or []

        self.requests = tornado.queues.Queue()
        self.responses = collections.defaultdict(list)

    async def worker(self):
        while True:
            await self.consume()

    async def validate(self, response):
        for validator in self.validators:
            validator(response)

    async def save(self, response):
        self.responses[response.request.group].append(response.body.decode('utf-8'))

    async def consume(self):
        async for request in self.requests:
            try:
                response = await self.client.fetch(request, raise_error=False)

                await self.validate(response)
                await self.save(response)
            except ResponseError:
                if request.retries &lt; self.retries:
                    request.retries += 1
                    await self.requests.put(request)
            finally:
                self.requests.task_done()


    async def produce(self):
        for group, urls in self.groups.items():
            for url in urls:
                request = tornado.httpclient.HTTPRequest(url)
                request.group = group
                request.retries = 0
                await self.requests.put(request)

    async def fetch(self):
        await self.produce()

        for __ in range(self.concurrency):
            tornado.ioloop.IOLoop.current().spawn_callback(self.worker)

        await self.requests.join()



class MainHandler(tornado.web.RequestHandler):

    async def get(self):
        urls = []

        with open('urls') as f:  # mock
            for line in f:
                urls.append(line.strip())

        crawler = Crawler({'default': urls})

        await crawler.fetch()

        self.write(crawler.responses)


if __name__ == '__main__':
    app = tornado.web.Application(
        (tornado.web.url(r'/', MainHandler),), debug=True
    )
    app.listen(8000)

    tornado.ioloop.IOLoop.current().start()
</code></pre>

<p>It looks to me like most of the memory usage is devoted to <code>self.responses</code>. Since you seem to be ordering responses by "group" before writing them to a file, I can understand why you do it this way. One idea is to store them in a database (MySQL or MongoDB or whatever) with the "group" as column or field value in the database record.</p>

<p>The database might be the final destination of your data, or else it might be a temporary place to store the data until crawler.fetch completes. Then, query all the data from the database, ordered by "group", and write it to the file.</p>

<p>This doesn't <em>solve</em> the problem, it just means that the database process is responsible for most of your memory usage, instead of the Python process. This may be preferable for you, however.</p>

python tornado