python: multiple .dat's in multiple arrays
<p>I'm trying to sort some data into (np.)arrays and get stuck with a problem.</p>

<p>I have 1000 .dat files and I need to put the data from them in 1000 different arrays. Further, every array should contain data depend on coordinates [i] [j] [k] (this part I've done already and the code looks like this (this is kind of "short" version):</p>

<pre><code>with open('177500.dat', newline='') as csvfile:
    f = csv.reader(csvfile, delimiter=' ')
    for row in f:
        &lt;some code which works pretty good&gt;
    cV = [[[[] for k in range(kMax)] for j in range(jMax)] for i in range(iMax)] 

with open('177500.dat', newline='') as csvfile:
    f = csv.reader(csvfile, delimiter=' ')
    &lt;some code which works also good&gt;
            values = np.array([np.float64(row[i]) for i in range(3, rowLen)])
            cV[int(row[0])][int(row[1])][int(row[2])] = values  
</code></pre>

<p>After this, i can print cV [i] [j] [k] and I get all data which is contained in one .dat file at the coordinates [i] [j] [k].</p>

<p>And now I need to create cV [i] [j] [k] <strong>[n]</strong> to get the data from the specific file number <strong>n</strong> at the coordinates [i] [j] [k]. And I absolutely don't know how can I tell python to put the data into the "right" place.</p>

<p>I tried some things like this:</p>

<pre><code>for m in range(160000,182501,2500):
with open ('output/%d.dat' % m, newline='') as csvfile:
  &lt;bla bla code&gt;
cV = [[[[[] for k in range(kMax)] for j in range(jMax)] for i in range(iMax)] for n in range(tMax)] 

if len(row) == rowLen:
            values = [np.array([np.float64(row[i]) for i in range (3, rowLen)]) for n in range(tMax)]
            for n in range(tMax):
                cV[int(row[0])][int(row[1])][int(row[2])][int(n)] = values[n]
</code></pre>

<p>But this surely didn't work because python don't know what the hack should be this <strong>[n]</strong> after the values.</p>

<p>So, how can I tell pyhton to put this [i] [j] [k] data from the file nr. <strong>n</strong> in the array cV [i] [j] [k] <strong>[n]</strong>?</p>

<p>Thanks in advance</p>

<p>C.</p>

<p>P.S. I didn't post the whole code because I don't think it is necessary. All arrays are created properly, but the thing which isn't working ist the data in them.</p>

<p>I think building arrays like this is going to make things more complicated for you. It would be easier to build a dictionary using tuples as keys. In the example file you sent me, each <code>(x, y, z)</code> pair was repeated twice, making me think that each file contains data on <em>two</em> iterations of a total solution of 2000 iterations. Dictionaries must have unique keys, so for each file I have implemented another counter, <code>timestep</code>, that can increment when collating data from a single file.</p>

<p>Now, if I wanted coords (1, 2, 3) on the 3rd timestep, I could do <code>simulation[(1, 2, 3, 3)]</code>.</p>

<pre><code>import csv
import numpy as np

'''
Made the assumptions that:
    -Each file contains two iterations from a simulation of 2000 iterations
    -Each file is numbered sequentially. Each time the same (x, y, z) coords are 
    discovered, it represents the next timestep in simulation

Accessing data is via a tuple key (x, y, z, n) with n being timestep
'''

simulation = {}
file_count = 1
timestep = 1
num_files = 2

for x in range(1, num_files + 1):
    with open('sim_file_{}.dat'.format(file_count), 'r') as infile:
        second_read = False
        reader = csv.reader(infile, delimiter=' ')
        for row in reader:
            item = [float(x) for x in row]
            if row:
                if (not second_read and not
                    any(simulation.get((item[0], item[1], item[2], timestep), []))):
                    timestep += 1
                    second_read = True
                simulation[(item[0], item[1], item[2], timestep)] = np.array(item[3:])

    file_count += 1
    timestep += 1
    second_read = False
</code></pre>

python arrays