urllib.request: Data Not Writing to Outfile
<p>I've got a script here which (ideally) iterates through multiple pages X of JSON data for each entity Y (in this case, multiple loans X for each team Y). The way that the api is constructed, I believe I must physically change a subdirectory within the URL in order to iterate through multiple entities. Here is the explicit documentation and URL:</p>

<blockquote>
  <p>GET /teams/:id/loans </p>
  
  <p>Returns loans belonging to a particular team.
  Example <a href="http://api.kivaws.org/v1/teams/2/loans.json" rel="nofollow">http://api.kivaws.org/v1/teams/2/loans.json</a></p>
  
  <p>Parameters id(number) Required. The team ID for which to return loans.
  page(number) The page position of results to return. Default: 1
  sort_by(string) The order by which to sort results. One of: oldest,
  newest Default: newest app_id(string) The application id in reverse
  DNS notation. ids_only(string) Return IDs only to make the return
  object smaller. One of: true, false Default: false Response<br>
  loan_listing â HTML , JSON , XML , RSS</p>
  
  <p>Status   Production</p>
</blockquote>

<p>And here is my script, which does run and appear to extract the correct data, but <strong>doesn't seem to write any data to the outfile:</strong></p>

<pre><code># -*- coding: utf-8 -*-

import urllib.request as urllib
import json
import time

# storing team loans dict. The key is the team id, en value is the list of lenders
team_loans = {}

url = "http://api.kivaws.org/v1/teams/"

#teams_id range 1 - 11885
for i in range(1, 100):

  params = dict(
    id = i
  )

  #i =1
  try:
    handle = urllib.urlopen(str(url+str(i)+"/loans.json"))
    print(handle)
  except:
    print("Could not handle url")
    continue
  # reading response
  item_html =  handle.read().decode('utf-8')
  # converting bytes to str
  data = str(item_html)
  # converting to json
  data = json.loads(data)
  # getting number of pages to crawl
  numPages = data['paging']['pages']
  # deleting paging data
  data.pop('paging')

  # calling additional pages
  if numPages &gt;1:
    for pa in range(2,numPages+1,1):
        #pa = 2
        handle = urllib.urlopen(str(url+str(i)+"/loans.json?page="+str(pa)))
        print("Pulling loan data from team " + str(i) + "...")
        # reading response
        item_html =  handle.read().decode('utf-8')
        # converting bytes to str
        datatemp = str(item_html)
        # converting to json
        datatemp = json.loads(datatemp)
        #Pagings are redundant headers
        datatemp.pop('paging')
        # adding data to initial list
        for loan in datatemp['loans']:
            data['loans'].append(loan)
        time.sleep(2)

  # recording loans by team in dict
  team_loans[i] = data['loans']
  if (data['loans']):
    print("===Data added to the team_loan dictionary===")
  else:
    print("!!!FAILURE to add data to team_loan dictionary!!!")

  # recorging data to file when 10 teams are read
  print("===Finished pulling from page " + str(i) + "===")
  if (int(i) % 10 == 0):
    outfile = open("team_loan.json", "w")
    print("===Now writing data to outfile===")
    json.dump(team_loans, outfile, sort_keys = True, indent = 2, ensure_ascii=True)
    outfile.close()
  else:
    print("!!!FAILURE to write data to outfile!!!")

  # compliance with API # of requests
  time.sleep(2)

print ('Done! Check your outfile (team_loan.json)')
</code></pre>

<p>I know that may be a heady amount of code to throw in your faces, but it's a pretty sequential process.</p>

<p>Again, this program is pulling the correct data, but it is <strong>not</strong>  writing this data to the outfile. Can anyone understand why? </p>

<p>For others who may read this post, the script does in face write data to an outfile. It was simply test code logic that was wrong. Ignore the print statements I have put into place.</p>

python json rest python-3.x urllib