Python how does == work for float/double?
<p>I know using <code>==</code> for float is generally not safe. But does it work for the below scenario?</p>

<ol>
<li>Read from csv file A.csv, save first half of the data to csv file B.csv without doing anything.</li>
<li>Read from both A.csv and B.csv. Use <code>==</code> to check if data match everywhere in the first half.</li>
</ol>

<p>These are all done with Pandas. The columns in A.csv have types datetime, string, and float. Obviously <code>==</code> works for datetime and string, so if <code>==</code> works for float as well in this case, it saves a lot of work.</p>

<p>It seems to be working for all my tests, but can I assume it will work all the time?</p>

<p>No, you cannot assume that this will work all the time.</p>

<p>For this to work, you need to know that the text value written out by Pandas when it's writing to a CSV file recovers the exact same value when read back in (again using Pandas). But by default, the Pandas <code>read_csv</code> function sacrifices accuracy for speed, and so the parsing operation does <em>not</em> automatically recover the same float.</p>

<p>To demonstrate this, try the following: we'll create some random values, write them out to a CSV file, and read them back in, all using Pandas. First the necessary imports:</p>

<pre><code>&gt;&gt;&gt; import pandas as pd
&gt;&gt;&gt; import numpy as np
</code></pre>

<p>Now create some random values, and put them into a Pandas <code>Series</code> object:</p>

<pre><code>&gt;&gt;&gt; test_values = np.random.rand(10000)
&gt;&gt;&gt; s = pd.Series(test_values, name='test_values')
</code></pre>

<p>Now we use the <code>to_csv</code> method to write these values out to a file, and then read the contents of that file back into a <code>DataFrame</code>:</p>

<pre><code>&gt;&gt;&gt; s.to_csv('test.csv', header=True)
&gt;&gt;&gt; df = pd.read_csv('test.csv')
</code></pre>

<p>Finally, let's extract the values from the relevant column of <code>df</code> and compare. We'll sum the result of the <code>==</code> operation to find out how many of the <code>10000</code> input values were recovered exactly.</p>

<pre><code>&gt;&gt;&gt; sum(test_values == df['test_values'])
7808
</code></pre>

<p>So approximately 78% of the values were recovered correctly; the others were not.</p>

<p>This behaviour is considered a feature of Pandas, rather than a bug. However, there's a workaround: Pandas 0.15 added a new <code>float_precision</code> argument to the CSV reader. By supplying <code>float_precision='round_trip'</code> to the <code>read_csv</code> operation, Pandas uses a slower but more accurate parser. Trying that on the example above, we get the values recovered perfectly:</p>

<pre><code>&gt;&gt;&gt; df = pd.read_csv('test.csv', float_precision='round_trip')
&gt;&gt;&gt; sum(test_values == df['test_values'])
10000
</code></pre>

<p>Here's a second example, going in the other direction. The previous example showed that writing and then reading doesn't give back the same data. This example shows that reading and then writing doesn't preserve the data, either. The setup closely matches the one you describe in the question. First we'll create <code>A.csv</code>, this time using regularly-spaced values instead of random ones:</p>

<pre><code>&gt;&gt;&gt; import pandas as pd, numpy as np
&gt;&gt;&gt; s = pd.Series(np.arange(10**4) / 1e3, name='test_values')
&gt;&gt;&gt; s.to_csv('A.csv', header=True)
</code></pre>

<p>Now we read <code>A.csv</code>, and write the first half of the data back out again to <code>B.csv</code>, as in your Step 1.</p>

<pre><code>&gt;&gt;&gt; recovered_s = pd.read_csv('A.csv').test_values
&gt;&gt;&gt; recovered_s[:5000].to_csv('B.csv', header=True)
</code></pre>

<p>Then we read in both <code>A.csv</code> and <code>B.csv</code>, and compare the first half of <code>A</code> with <code>B</code>, as in your Step 2.</p>

<pre><code>&gt;&gt;&gt; a = pd.read_csv('A.csv').test_values
&gt;&gt;&gt; b = pd.read_csv('B.csv').test_values
&gt;&gt;&gt; (a[:5000] == b).all()
False
&gt;&gt;&gt; (a[:5000] == b).sum()
4251
</code></pre>

<p>So again, several of the values don't compare correctly. Opening up the files, <code>A.csv</code> looks pretty much as I expect. Here are the first 15 entries in <code>A.csv</code>:</p>

<pre><code>,test_values
0,0.0
1,0.001
2,0.002
3,0.003
4,0.004
5,0.005
6,0.006
7,0.007
8,0.008
9,0.009
10,0.01
11,0.011
12,0.012
13,0.013
14,0.014
15,0.015
</code></pre>

<p>And here are the corresponding entries in <code>B.csv</code>:</p>

<pre><code>,test_values
0,0.0
1,0.001
2,0.002
3,0.003
4,0.004
5,0.005
6,0.006
7,0.006999999999999999
8,0.008
9,0.009000000000000001
10,0.01
11,0.011000000000000001
12,0.012
13,0.013000000000000001
14,0.013999999999999999
15,0.015
</code></pre>

<p>See this <a href="https://github.com/pydata/pandas/issues/8002" rel="nofollow">bug report</a> for more information on the introduction of the <code>float_precision</code> keyword to <code>read_csv</code>.</p>

python pandas floating-point