Pandas: when can I directly assign to values array
<p>I was messing with pandas in the interpreter and the following behavior took me by surprise:</p>

<pre><code>&gt;&gt;&gt; data2 = [[1, np.nan], [2, -17]]
&gt;&gt;&gt; f2 = pd.DataFrame(data2)
&gt;&gt;&gt; f2
   0     1
0  1   NaN
1  2 -17.0
&gt;&gt;&gt; f2.values[1, 1] = -99.0
&gt;&gt;&gt; f2
   0     1
0  1   NaN
1  2 -17.0
&gt;&gt;&gt; type(f2.values[0, 0]), type(f2.values[1, 0])
(&lt;class 'numpy.float64'&gt;, &lt;class 'numpy.float64'&gt;)
</code></pre>

<p>I am unable to assign directly to the underlying array using the values attribute.  However, if I explicitly start with floats, I can:</p>

<pre><code>&gt;&gt;&gt; data = [[1.0, np.nan], [2.0, -17.0]]
&gt;&gt;&gt; f = pd.DataFrame(data)
&gt;&gt;&gt; f
     0     1
0  1.0   NaN
1  2.0 -17.0
&gt;&gt;&gt; f.values[1, 1] = -99.0
&gt;&gt;&gt; f
     0     1
0  1.0   NaN
1  2.0 -99.0
</code></pre>

<p>Does anyone know a rule that would have allowed me to predict this?  I feel like I must be missing something obvious.</p>

<p>Pandas does not guarantee when assignments to <code>df.values</code> affect <code>df</code>, so I would recommend <em>never</em> trying to modify <code>df</code> via <code>df.values</code>. How and when this works is an implementation detail. As <a href="http://stackoverflow.com/a/39693521/190597">StevenG states</a>, use <code>df.iloc[1,1] = -99</code> to assign a new value at a location specified by ordinal indices, or use <code>df.set_value</code> to assign a new value at a location specified by index labels.</p>

<p>Under the hood, <code>df</code> stores its values in "blocks". The blocks are segregated by
dtype, though sometimes more than one block can have the same dtype. The data in
each block is stored in a NumPy array.</p>

<p>When you use</p>

<pre><code>df2 = pd.DataFrame([[1, np.nan], [2, -17]])
</code></pre>

<p>The first column has integer dtype while the second column has floating point dtype.</p>

<pre><code>In [27]: df2.info()
&lt;class 'pandas.core.frame.DataFrame'&gt;
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
0    2 non-null int64
1    1 non-null float64
dtypes: float64(1), int64(1)
memory usage: 112.0 bytes
</code></pre>

<p>When you access the <code>df2.values</code> property, a <em>single</em> NumPy array of one homogenous dtype is returned. When
the <code>df2</code> has columns of different dtypes, Pandas must promote the dtypes to a
single common dtype. In the worst case, the common dtype may be <code>object</code>. In this
case, the integers are promoted to floating point dtype.  </p>

<pre><code>In [28]: df2.values.dtype
Out[28]: dtype('float64')
</code></pre>

<p>The dtype promotion requires that the underlying data from the various blocks be
<em>copied</em> into a new NumPy array. Thus, modifying the copy returned by
<code>df2.values</code> does not affect the original data in <code>df2</code>.</p>

<p>In contrast, if the DataFrame's data is entirely of one dtype, then <code>f.values</code>
returns a view of the original data. So in this special case, modifying
<code>f.values</code> affects <code>f</code> itself.</p>

<hr>

<p>To summarize, when a DataFrame contains multiple blocks, <code>df.values</code> will be a
copy of the data in <code>df</code>. Modifying the copy in <code>df.values</code> will not affect
<code>df</code>.</p>

<p>Since a DataFrame can sometimes have multiple blocks of the <em>same dtype</em>, even
if all the data in the DataFrame has the same dtype, <code>df.values</code> may still be a
copy.</p>

<p>If you call <a href="https://github.com/pydata/pandas/blob/master/pandas/core/generic.py#L2802" rel="nofollow"><code>df.consolidate()</code></a> then data of each dtype will be grouped together
in a single NumPy array. So if your DataFrame's data consists of a single dtype,
and you call <code>df.consolidate()</code> first, then modifying <code>df.values</code> will modify
<code>df</code>.</p>

python pandas