Class for making asynchronous API requests with Twisted
<p>I am working on development of a system that collect data from rest servers and manipulates it.</p>

<p>One of the requirements is multiple and frequent API requests. We currently implement this in a somewhat synchronous way. I can easily implement this using threads but considering the system might need to support thousands of requests per second I think it would be wise to utilize Twisted's ability to efficiently implement the above. I have seen <a href="http://oubiwann.blogspot.co.il/2008/06/async-batching-with-twisted-walkthrough.html" rel="nofollow">this blog post</a> and the whole idea of deferred list seems to do the trick. But I am kind of stuck with how to structure my class (can't wrap my mind around how Twisted works).</p>

<p>Can you try to outline the structure of the class that will run the event-loop and will be able to get a list of URLs and headers and return a list of results after making the requests asynchronously?</p>

<p>Do you know of a better way of implementing this in python?</p>

<p>Sounds like you want to use a Twisted project called <a href="http://treq.readthedocs.io/en/latest/index.html" rel="nofollow"><code>treq</code></a> which allows you to send requests to HTTP endpoints. It works alot like <a href="http://docs.python-requests.org/en/master/" rel="nofollow"><code>requests</code></a>.  I recently helped a friend <a href="http://stackoverflow.com/questions/39133507/writing-a-twisted-client-to-send-looping-get-request-to-multiple-api-calls-and-r/39136513#39136513">here in this thread</a>.  My answer there might be of some use to you.  If you still need more help, just make a comment and I'll try my best to update this answer.</p>

python asynchronous twisted