How does numpy handle data files with uncertainty values, e.g., 0.6499(6)?
<p>Here is a snippet of a large data set I am working with:</p>

<pre><code># p*  T*     P*         U*          P*_cs    U*_cs  Steps  dt*
0.1   6.0    0.6499(6) -0.478(2)    0.6525  -0.452  30000  0.002
0.2   6.0    1.442(1)  -0.942(2)    1.452   -0.890  30000  0.002
0.3   6.0    2.465(3)  -1.376(1)    2.489   -1.298  30000  0.002
0.4   6.0    3.838(5)  -1.785(3)    3.880   -1.681  20000  0.002
0.5   6.0    5.77(1)   -2.131(3)    5.84    -2.000  20000  0.002
0.6   6.0    8.51(2)   -2.382(5)    8.60    -2.225  20000  0.002
0.7   6.0   12.43(2)   -2.501(4)   12.56    -2.318  20000  0.002
0.8   6.0   18.05(2)   -2.416(4)   18.22    -2.207  20000  0.002
0.9   6.0   26.00(2)   -2.058(4)   26.21    -1.823  20000  0.004
1.0   6.0   37.06(3)   -1.361(6)   37.32    -1.100  20000  0.002
1.1   6.0   52.25(2)   -0.216(4)   52.57     0.072  20000  0.002
1.2   6.0   72.90(5)    1.502(9)   73.28     1.816  20000  0.002
1.25  6.0   85.71(5)    2.612(8)   86.12     2.939  20000  0.002
</code></pre>

<p>Loading in this data set using <code>np.loadtxt</code> fails because of the uncertainties for the P* and U* values.  Is there a built-in tool for handling this to avoid manually editing the data files? </p>

<p>I am looking at the <a href="https://pythonhosted.org/uncertainties/numpy_guide.html" rel="nofollow">uncertainties</a> package as a possible solution but I wonder if numpy already has something for this.</p>

<pre><code>In [1]: txt=b"""# p*  T*     P*         U*          P*_cs    U*_cs  Steps  dt*
   ...: 0.1   6.0    0.6499(6) -0.478(2)    0.6525  -0.452  30000  0.002
   ...: 0.2   6.0    1.442(1)  -0.942(2)    1.452   -0.890  30000  0.002
   ...: 0.3   6.0    2.465(3)  -1.376(1)    2.489   -1.298  30000  0.002"""
In [2]: txt=txt.splitlines()
</code></pre>

<p><code>txt</code> is a file substitue (bytestring in PY3)</p>

<pre><code>In [3]: data=np.genfromtxt(txt, dtype=None, names=True)
In [4]: data
Out[4]: 
array([(0.1, 6.0, b'0.6499(6)', b'-0.478(2)', 0.6525, -0.452, 30000, 0.002),
       (0.2, 6.0, b'1.442(1)', b'-0.942(2)', 1.452, -0.89, 30000, 0.002),
       (0.3, 6.0, b'2.465(3)', b'-1.376(1)', 2.489, -1.298, 30000, 0.002)], 
      dtype=[('p', '&lt;f8'), ('T', '&lt;f8'), ('P', 'S9'), ('U', 'S9'), ('P_cs', '&lt;f8'), ('U_cs', '&lt;f8'), ('Steps', '&lt;i4'), ('dt', '&lt;f8')])
</code></pre>

<p>'P' and 'U' are loaded as strings because they can't be parsed as numbers.</p>

<p>Now define a <code>converter</code> that strips off the <code>()</code> part (again with bytestrings)</p>

<pre><code>def rmvpar(astr):
   return float(astr.split(b'(')[0])

In [9]: data=np.genfromtxt(txt, dtype=None, names=True, 
      converters={2:rmvpar, 3:rmvpar})
In [10]: data
Out[10]: 
array([(0.1, 6.0, 0.6499, -0.478, 0.6525, -0.452, 30000, 0.002),
       (0.2, 6.0, 1.442, -0.942, 1.452, -0.89, 30000, 0.002),
       (0.3, 6.0, 2.465, -1.376, 2.489, -1.298, 30000, 0.002)], 
      dtype=[('p', '&lt;f8'), ('T', '&lt;f8'), ('P', '&lt;f8'), ('U', '&lt;f8'), ('P_cs', '&lt;f8'), ('U_cs', '&lt;f8'), ('Steps', '&lt;i4'), ('dt', '&lt;f8')])
</code></pre>

<p>Now those two fields are floats.</p>

<p>But converters can't return two numbers, so I can't keep the uncertainty this way.</p>

<p>Another approach is to pass the lines through a filter function</p>

<pre><code>def splt(astr):
    strs=astr.split()
    def foo(astr):
        if b'(' in astr:
            astr = astr.strip(b')').split(b'(')
            return b','.join(astr)
        return astr
    return b','.join([foo(a) for a in strs])

In [26]: [splt(line) for line in txt]
Out[26]: 
[b'#,p*,T*,P*,U*,P*_cs,U*_cs,Steps,dt*',
 b'0.1,6.0,0.6499,6,-0.478,2,0.6525,-0.452,30000,0.002',
 b'0.2,6.0,1.442,1,-0.942,2,1.452,-0.890,30000,0.002',
 b'0.3,6.0,2.465,3,-1.376,1,2.489,-1.298,30000,0.002']
</code></pre>

<p>To use this I have to skip the header because the new lines have two added columns</p>

<pre><code>In [28]: data=np.genfromtxt([splt(line) for line in txt], delimiter=',',dtype=None, skip_header=1)
In [29]: data
Out[29]: 
array([(0.1, 6.0, 0.6499, 6, -0.478, 2, 0.6525, -0.452, 30000, 0.002),
       (0.2, 6.0, 1.442, 1, -0.942, 2, 1.452, -0.89, 30000, 0.002),
       (0.3, 6.0, 2.465, 3, -1.376, 1, 2.489, -1.298, 30000, 0.002)], 
      dtype=[('f0', '&lt;f8'), ('f1', '&lt;f8'), ('f2', '&lt;f8'), ('f3', '&lt;i4'), 
             ('f4', '&lt;f8'), ('f5', '&lt;i4'), ('f6', '&lt;f8'), ('f7', '&lt;f8'), 
             ('f8', '&lt;i4'), ('f9', '&lt;f8')])
</code></pre>

<p>But I could modify the original <code>dtype</code> to make the 2 fields (sub)arrays:</p>

<pre><code>In [30]: dt=np.dtype([('p', '&lt;f8'), ('T', '&lt;f8'), ('P', '&lt;f8',(2,)), 
                ('U', '&lt;f8',(2,)), ('P_cs', '&lt;f8'), ('U_cs', '&lt;f8'), 
                ('Steps', '&lt;i4'), ('dt', '&lt;f8')])

In [31]: data = np.genfromtxt((splt(line) for line in txt), delimiter=',',dtype=dt, skip_header=1)
In [32]: data
Out[32]: 
array([(0.1, 6.0, [0.6499, 6.0], [-0.478, 2.0], 0.6525, -0.452, 30000, 0.002),
       (0.2, 6.0, [1.442, 1.0], [-0.942, 2.0], 1.452, -0.89, 30000, 0.002),
       (0.3, 6.0, [2.465, 3.0], [-1.376, 1.0], 2.489, -1.298, 30000, 0.002)], 
      dtype=[('p', '&lt;f8'), ('T', '&lt;f8'), ('P', '&lt;f8', (2,)), ('U', '&lt;f8', (2,)), 
             ('P_cs', '&lt;f8'), ('U_cs', '&lt;f8'), ('Steps', '&lt;i4'), ('dt', '&lt;f8')])
</code></pre>

<p>Such a field would look like:</p>

<pre><code>In [33]: data['P']
Out[33]: 
array([[ 0.6499,  6.    ],
       [ 1.442 ,  1.    ],
       [ 2.465 ,  3.    ]])
</code></pre>

<p>I could define other <code>dtypes</code>, just as long as the number of fields match.</p>

<p>With a file, rather than these text lines, I would use something like (not tested):</p>

<pre><code>with open(filename,'wb') as f:
    data = np.genfromtxt((splt(line) for line in f),...
</code></pre>

<p>Here, and above, I'm using the generator expression <code>(splt(line) for line in x)</code>, though the list comprehension would be fine.  Any code that opens the file and yields/returns the modified lines will work.</p>

python numpy uncertainty