How to write large JSON data?
<p>I have been trying to write large amount (>800mb) of data to JSON file; I did some fair amount of trial and error to get this code:</p>

<pre><code>def write_to_cube(data):
    with open('test.json') as file1:
        temp_data = json.load(file1)

    temp_data.update(data)

    file1.close()

    with open('test.json', 'w') as f:
        json.dump(temp_data, f)

        f.close()
</code></pre>

<p>to run it just call the function <code>write_to_cube({"some_data" = data})</code></p>

<p>Now the problem with this code is that it's fast for the small amount of data, but the problem comes when <code>test.json</code> file has more than 800mb in it. When I try to update or add data to it, it takes ages.</p>

<p>I know there are external libraries such as <code>simplejson</code> or <code>jsonpickle</code>, I am not pretty sure on how to use them.</p>

<p>Is there any other way to this problem?</p>

<p><strong>Update:</strong></p>

<p>I am not sure how this can be a duplicate, other articles say nothing about writing or updating a large JSON file, rather they say only about parsing.</p>

<p><a href="http://stackoverflow.com/questions/2400643/is-there-a-memory-efficient-and-fast-way-to-load-big-json-files-in-python">Is there a memory efficient and fast way to load big json files in python?</a></p>

<p><a href="http://stackoverflow.com/questions/10382253/reading-rather-large-json-files-in-python">Reading rather large json files in Python</a></p>

<p>None of the above resolve this question a duplicate. They don't say anything about writing or update.</p>

<p>So the problem is that you have a long operation. Here are a few approaches that I usually do:</p>

<ol>
<li>Optimize the operation: This rarely works. I wouldn't recommend any superb library that would parse the json a few seconds faster</li>
<li>Change your logic: If the purpose is to save and load data, probably you would like to try something else, like storing your object into a binary file instead.</li>
<li>Threads and callback, or deferred objects in some web frameworks. In case of web applications, sometimes, the operation takes longer than a request can wait, we can do the operation in background (some cases are: zipping files, then send the zip to user's email, sending SMS by calling another third party's api...)</li>
</ol>

python json