Convert import string to float with numpy's loadtext
<p>I'm attempting to import text from a flat file and to convert it to float values within a single line. I've seen <a href="http://stackoverflow.com/questions/8420143/valueerror-could-not-convert-string-to-float-id">this post</a> which has the same error, but I haven't found which characters are invalid in my input file. Or do I have a syntax error?</p>

<p>Import as a string an print the result:</p>

<pre><code>data = np.loadtxt(file, delimiter='\t', dtype=str)
print(data[0:2])
... 
[["b'Time'" "b'Percent'"]
 ["b'99'" "b'0.067'"]]
</code></pre>

<p>Attempt to import as float:</p>

<pre><code># Import data as floats and skip the first row: data_float
data_float = np.loadtxt(data, delimiter='\t', dtype=float, skiprows=1)
</code></pre>

<p>It throws the following error:</p>

<pre><code>Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    data_float = np.loadtxt(data, delimiter='\t', dtype=float, skiprows=1)
  File "&lt;stdin&gt;", line 848, in loadtxt
    items = [conv(val) for (conv, val) in zip(converters, vals)]
  File "&lt;stdin&gt;", line 848, in &lt;listcomp&gt;
    items = [conv(val) for (conv, val) in zip(converters, vals)]
ValueError: could not convert string to float: b'["b\'99\'" "b\'0.067\'"]'
</code></pre>

<p>By the way, I've also seen <a href="http://stackoverflow.com/questions/6269765/what-does-the-b-character-do-in-front-of-a-string-literal">this post</a> which explains the <code>b</code> character, but I don't think that's the issue.</p>

<p>An additional troubleshooting step as suggested by the first answer:</p>

<pre><code>data = np.loadtxt(file, delimiter="\tb'", dtype=str)
</code></pre>

<p>Returns:</p>

<pre><code>array(["b'Time\\tPercent'", "b'99\\t0.067'", "b'99\\t0.133'",
       "b'99\\t0.067'", "b'99\\t0'", "b'99\\t0'", "b'0\\t0.5'",
       "b'0\\t0.467'", "b'0\\t0.857'", "b'0\\t0.5'", "b'0\\t0.357'",
       "b'0\\t0.533'", "b'5\\t0.467'", "b'5\\t0.467'", "b'5\\t0.125'",
       "b'5\\t0.4'", "b'5\\t0.214'", "b'5\\t0.4'", "b'10\\t0.067'",
       "b'10\\t0.067'", "b'10\\t0.333'", "b'10\\t0.333'", "b'10\\t0.133'",
       "b'10\\t0.133'", "b'15\\t0.267'", "b'15\\t0.286'", "b'15\\t0.333'",
       "b'15\\t0.214'", "b'15\\t0'", "b'15\\t0'", "b'20\\t0.267'",
       "b'20\\t0.2'", "b'20\\t0.267'", "b'20\\t0.437'", "b'20\\t0.077'",
       "b'20\\t0.067'", "b'25\\t0.133'", "b'25\\t0.267'", "b'25\\t0.412'",
       "b'25\\t0'", "b'25\\t0.067'", "b'25\\t0.133'", "b'30\\t0'",
       "b'30\\t0.071'", "b'30\\t0'", "b'30\\t0.067'", "b'30\\t0.067'",
       "b'30\\t0.133'"], 
      dtype='&lt;U16')
</code></pre>

<p>The problem is that your numbers are quoted. That is, the field is <code>'99'</code>, rather than <code>99</code>. There are two ways you can do this. You can provide converter functions that strip the quotes and return a float. Or you can use the <code>csv</code> module to load your data in and then pass that data to <code>numpy</code>.</p>

<p>Using converter functions</p>

<pre><code>import numpy as np
from io import StringIO

data = """'x'\t'y'
'1'\t'2.5'"""

arr = np.loadtxt(StringIO(data), dtype=float, delimiter="\t", skiprows=1, 
    converters=dict.fromkeys([0, 1], (lambda s: float(s.strip(b"'"))))
)
</code></pre>

<p>Using <code>csv</code></p>

<pre><code>import csv
import numpy as np
from io import StringIO

data = """'x'\t'y'
'1'\t'2.5'"""

reader = csv.reader(StringIO(data), quotechar="'", delimiter="\t")
next(reader) # skip headers
arr = np.array(list(reader), dtype=float)
</code></pre>

<p>In both examples I've uses <code>StringIO</code> so you can easily see the contents of the "file". You can of course pass the filename or file object to these functions.</p>

python python-3.x ipython