Split large JSON file into batches of 100 at a time to run through an API
<p>I am so close to being done with this tool I am developing, but as a junior developer with NO senior programmer to work with I am stuck. I have a script in python that takes data from our data base converts it to JSON to be run through an Address validation API, I have it all working, but the fact is that the API only accepts 100 objects at a time. I need to basically break up the file with X objects into batches of 100 to be run then stored into the same output file. Here is the snippit of my script structure:</p>

<pre><code>for row in rows:

    d = collections.OrderedDict()
    d['input_id'] = str(row.INPUT_ID)
    d['addressee'] = row.NAME
    d['street'] = row.ADDRESS
    d['city'] = row.CITY
    d['state'] = row.STATE
    d['zipcode'] = row.ZIP
    d['candidates'] = row.CANDIDATES
    obs_list.append(d)

json.dump(obs_list, file)

ids_file = '.csv'

cur.execute(input_ids)

columns = [i[0] for i in cur.description]

ids_input = cur.fetchall()

#ids_csv = csv.writer(with open('.csv','w',newline=''))

with open('.csv','w',newline='') as f:

    ids_csv = csv.writer(f,delimiter=',')

    ids_csv.writerow(columns)

    ids_csv.writerows(ids_input)

print('Run through API')

url = 'https://api.'

headers = {'content-type': 'application/json'}
</code></pre>

<p>this  is where i assume i need to do the loop to break it up  </p>

<pre><code>with open('.json', 'r') as run:

    dict_run = run.readlines()

    dict_ready = (''.join(dict_run))

#lost :(
    for object in dict_ready:

        # do something with object to only run 100 at a time

        r = requests.post(url, data=dict_ready, headers=headers)

ss_output = r.text

output = 'C:\\Users\\TurnerC1\\Desktop\\ss_output.json'

with open(output,'w') as of:

    of.write(ss_output)
</code></pre>

<p>at the moment I have about 4,000 of these in a file to be run through the API that only accepts 100 at a time. Im sure there is an easy answer, I am just burnt out doing this by myself lol. Any help is greatly appreciated. </p>

<p>sample json:</p>

<pre><code>[
        {
            "street":"1 Santa Claus",
            "city":"North Pole",
            "state":"AK",
            "candidates":10
        },
        {
            "addressee":"Apple Inc",
            "street":"1 infinite loop",
            "city":"cupertino",
            "state":"CA",
            "zipcode":"95014",
            "candidates":10
        }
    ]'
</code></pre>

<p>As I read it, you have a range of objects, and you want to break it up into subsets, each of which is no more than 100 items.</p>

<p>Assuming that <code>dict_ready</code> is a list of objects (if it's not, modify your code to make it so):</p>

<pre><code>count = 100
subsets = (dict_ready[x:x + count] for x in range(0, len(dict_ready), count))
for subset in subsets:
    r = requests.post(url, data=subset, headers=headers)
</code></pre>

python json