Extracting articles from New York Post by using Python and New York Post API
<p>I am trying to create a corpus of text documents via the New york Times API (articles concerning terrorist attacks) on Python.</p>

<p>I am aware that the NYP API do not provide the full body text, but provides the URL from which I can scrape the article. So the idea is to extract the "web_url" parameters from the API and consequently scrape the full body article.</p>

<p>I am trying to use the NYT API library on Python with these lines:</p>

<pre><code>from nytimesarticle import articleAPI

api = articleAPI("*Your Key*")
articles = api.search( q = 'terrorist attack')

print(articles['response'],['docs'],['web_url'])
</code></pre>

<p>But I cannot extract the "web_url" or the articles. All I get is this output:</p>

<pre><code>{'meta': {'time': 19, 'offset': 10, 'hits': 0}, 'docs': []} ['docs'] ['web_url']
</code></pre>

<p>The comma in the print statement separates what is printed. </p>

<p>You'll want something like this </p>

<pre><code>articles['response']['docs']['web_url']
</code></pre>

<p>But <code>'docs': []</code> is both an array and empty, so above line won't work, so you could try </p>

<pre><code>articles = articles['response']['docs']
for article in articles:
    print(article['web_url']) 
</code></pre>

python