I have a list of IDs, each of which is again associated with several IDS and some values. How to code in python to save this data?
<p>I am reading in data of the following type from a file, and I need a method to store it for further calculations.</p>

<p>ID1    ,      ID2       ,    value</p>

<p>A      ,      1         ,     520</p>

<p>A      ,      2         ,      180</p>

<p>A      ,      3         ,      80</p>

<p>B      ,      1         ,      49</p>

<p>C      ,      1        ,       96</p>

<p>C       ,     2          ,     287</p>

<p>etc.</p>

<p>What is the best way to save it?</p>

<p>In PERL, I would have used a hash and separators, as follows, and then called by hash key and separated using split over the comma:</p>

<p>$data{$ID1} .= $ID2.':'.$value.',';</p>

<p>I have to address the following problem in PYTHON as it would be integrated with other code, but I am new to the language. Please suggest what might be the best way to do it.</p>

<p>P.S. The input data file is huge (~500Mb) and could be more.</p>

<p>Thanks for the help.</p>

<p>If you've loaded your data with Python, and your next program down the line is also written in Python, you can simply use the <code>pickle</code> module, like this:</p>

<pre><code>big_list_list = [["A", 1, 520], ["A", 2, 180], ["B", 1, 49]]

import pickle

# Storing the data
with open("data.pickle", "wb") as outfile:
    pickle.dump(big_list_list, outfile)

# Retrieving the data
with with open("data.pickle", "rb") as infile:
    reconstructed_big_list_list = pickle.load(infile)
</code></pre>

<p>This has two caveats: if part of your workflow includes Non-Python programs, they won't be able to read pickles. And you shouldn't trust pickle files from arbitrary sources, since they could contain malicious code.</p>

<p>Instead of using pickles, you can also use JSON files. Simple replace the word <code>pickle</code> with <code>json</code> in the recipy above. JSON has the advantage that many Non-Python programs can deal with it.</p>

<p>Even more universal would be the use of CSV files, like this:</p>

<pre><code>import csv

with open('data.csv', 'w', newline='') as outfile:
    writer = csv.writer(outfile)
    writer.writerows(big_list_list)

with open('data.csv', newline='') as infile:
    reader = csv.reader(infile)
    reconstructed_big_list_list = [row for row in reader]
</code></pre>

<p>Python's standard library also includes the module <code>sqlite3</code>, which allows you to write your data to a database, which might be useful if your data becomes more complicated than a simple list of lists, or you need concurrent access.</p>

<p>PS.: I just saw that you noted that your files could be uge. In this case, you could modify the CSV solution to store and load your data incrementally:</p>

<pre><code>import csv

with open('data.csv', 'w', newline='') as outfile:
    writer = csv.writer(outfile)
    for row in big_list_list:
        writer.writerow(row)

with open('data.csv', newline='') as infile:
    reader = csv.reader(infile)
    for row in reader:
        print(row)
</code></pre>

python