Extracting articles from New York Post by using Python and New York Post API
<p>I am trying to create a corpus of text documents via the New york Times API (articles concerning terrorist attacks) on Python.</p>

<p>I am aware that the NYP API do not provide the full body text, but provides the URL from which I can scrape the article. So the idea is to extract the "web_url" parameters from the API and consequently scrape the full body article.</p>

<p>I am trying to use the NYT API library on Python with these lines:</p>

<pre><code>from nytimesarticle import articleAPI

api = articleAPI("*Your Key*")
articles = api.search( q = 'terrorist attack')

print(articles['response'],['docs'],['web_url'])
</code></pre>

<p>But I cannot extract the "web_url" or the articles. All I get is this output:</p>

<pre><code>{'meta': {'time': 19, 'offset': 10, 'hits': 0}, 'docs': []} ['docs'] ['web_url']
</code></pre>

<p>There seems to be an issue with the <code>nytimesarticle</code> module itself. For example, see the following:</p>

<pre><code>&gt;&gt;&gt; articles = api.search(q="trump+women+accuse", begin_date=20161001)
&gt;&gt;&gt; print(articles)
{'response': {'docs': [], 'meta': {'offset': 0, 'hits': 0, 'time': 21}}, 'status': 'OK', 'copyright': 'Copyright (c) 2013 The New York Times Company.  All Rights Reserved.'}
</code></pre>

<p>But if I use <a href="http://docs.python-requests.org" rel="nofollow"><code>requests</code></a> (as is used in the module) to access the API directly, I get the results I'm looking for:</p>

<pre><code>&gt;&gt;&gt; import requests
&gt;&gt;&gt; r = requests.get("http://api.nytimes.com/svc/search/v2/articlesearch.json?q=trump+women+accuse&amp;begin_date=20161001&amp;api-key=XXXXX")
&gt;&gt;&gt; data = r.json()
&gt;&gt;&gt; len(data["response"]["docs"])
10
</code></pre>

<p>meaning that 10 articles were returned (the full value of <code>data</code> is 16kb, so I won't include it all here). Contrast that to the response from <code>api.search()</code>, where <code>articles["response"]["docs"]</code> is an empty list.</p>

<p><code>nytimesarticle.py</code> is only 115 lines long, so it's pretty straightforward to debug. Printing the value of the URL sent to the API reveals this:</p>

<pre><code>&gt;&gt;&gt; articles = api.search(q="trump+women+accuse", begin_date=20161001)
https://api.nytimes.com/svc/search/v2/articlesearch.json?q=b'trump+women+accuse'&amp;begin_date=20161001&amp;api-key=XXXXX
#                                                          ^^ THIS
</code></pre>

<p>The <a href="https://github.com/evansherlock/nytimesarticle/blob/89f551699ffb11f71b47271246d350a1043e9326/nytimesarticle.py#L29-L42" rel="nofollow">offending code</a> encodes every string parameter to UTF-8, which makes it a <code>bytes</code> object. This is not necessary, and wrecks the constructed URL as shown above. Fortunately, there is a <a href="https://github.com/evansherlock/nytimesarticle/pull/2" rel="nofollow">pull request</a> that fixes this:</p>

<pre><code>&gt;&gt;&gt; articles = api.search(q="trump+women+accuse", begin_date=20161001)
http://api.nytimes.com/svc/search/v2/articlesearch.json?begin_date=20161001&amp;q=trump+women+accuse&amp;api-key=XXXXX
&gt;&gt;&gt; len(articles["response"]["docs"])
10
</code></pre>

<p>This also allows for other string parameters such as <code>sort="newest"</code> to be used, as the bytes formatting was causing an error previously.</p>

python