Defining columns in numpy array - TypeError: invalid type promotion
<p>I am defining an array which should look like this </p>

<pre><code>['word1', 2000, 21]
['word2', 2002, 33]
['word3', 1988, 51]
['word4', 1999, 26]
['word5', 2001, 72]
</code></pre>

<p>However when I append an a new entry I get a TypeError. </p>

<pre><code>import numpy as np

npdtype = [('word', 'S35'), ('year', int), ('wordcount', int)]
np_array = np.empty((0,3), dtype=npdtype)

word = 'word1'
year = '2001'
word_count = '21'

np_array = np.append(np_array, [['word1', int(year), int(word_count)]], axis=0)
</code></pre>

<p>Traceback </p>

<pre><code> File "/home/matt/.local/lib/python2.7/site-packages/numpy/lib/function_base.py", line 4586, in append
return concatenate((arr, values), axis=axis)
 TypeError: invalid type promotion
</code></pre>

<p>What am I doing wrong?</p>

<p>Thanks</p>

<p><code>append</code> is a way of calling <code>np.concatenate</code>.  Look at its code.  Note it has to make sure the 2nd argument is an array. It does that without knowledge of your special <code>dtype</code>.  Try that.  It probably produces a string dtype. Then it tries the concatenate.  So you need to make an array with the correct dtype first.</p>

<p>I discourage the use of <code>append</code>; it's better to use <code>concatenate</code> directly so you have understand all details.</p>

<p>======================</p>

<p>Expanding on your answer:</p>

<pre><code>In [75]: npdtype
Out[75]: [('word', 'S35'), ('year', numpy.int16), ('wordcount', numpy.int16)]
In [76]: column = np.array( [b'word1', np.int16(year), np.int16(word_count)], dtype=npdtype)
In [77]: column
Out[77]: 
array([(b'word1', 0, 0), 
       (b'\xd1\x07', 0, 0), 
       (b'\x15', 0, 0)], 
      dtype=[('word', 'S35'), ('year', '&lt;i2'), ('wordcount', '&lt;i2')])
</code></pre>

<p>I don't think this is what you want.</p>

<p>The correct way to provide data for structured array record is with a tuple, or list of tuples (note the extra ()):</p>

<pre><code>In [78]: column = np.array( [(b'word1', np.int16(year), np.int16(word_count))], dtype=npdtype)
In [79]: column
Out[79]: 
array([(b'word1', 2001, 21)], 
      dtype=[('word', 'S35'), ('year', '&lt;i2'), ('wordcount', '&lt;i2')])
In [80]: column.shape
Out[80]: (1,)
</code></pre>

<p>Now I have a 1d, 1 element array with 3 fields.</p>

<p>Without the [], I get a single element 0d array</p>

<pre><code>In [81]: column0 = np.array( (b'word1', np.int16(year), np.int16(word_count)), dtype=npdtype)
In [82]: column0.shape
Out[82]: ()
In [83]: column0
Out[83]: 
array((b'word1', 2001, 21), 
      dtype=[('word', 'S35'), ('year', '&lt;i2'), ('wordcount', '&lt;i2')])
</code></pre>

<p>I can concatenate several of the 1d arrays:</p>

<pre><code>In [85]: np.concatenate([column,column,column])
Out[85]: 
array([(b'word1', 2001, 21), 
       (b'word1', 2001, 21), 
       (b'word1', 2001, 21)], 
      dtype=[('word', 'S35'), ('year', '&lt;i2'), ('wordcount', '&lt;i2')])
In [86]: _.shape
Out[86]: (3,)
In [87]: __['year']   # access the 2nd field (not column)
Out[87]: array([2001, 2001, 2001], dtype=int16)
</code></pre>

<p>Regarding the need for <code>b</code>.  You are using Py3 (as I am), and unicode is the default string type.  So if you had used <code>U35</code> in <code>npdtype</code>, you could have left off the <code>b</code> (bytestring flag).</p>

<p>That <code>(0,3)</code> shape initial array is probably not what you want.  0 rows, 3 columns, but still has 3 dtype fields.  Look at a <code>(1,3)</code> version</p>

<pre><code>In [88]: np.empty((1,3),dtype=npdtype)
Out[88]: 
array([[(b'', 0, 0), (b'', 0, 0), (b'', 0, 0)]], 
      dtype=[('word', 'S35'), ('year', '&lt;i2'), ('wordcount', '&lt;i2')])
</code></pre>

<p>This has blanks and 0 because of what happens to be in the memory.  They could have been random characters/numbers.</p>

<p><code>numpy</code> lets you make arrays with one or more 0 dimensions, but they usually aren't useful.  About the only place they appear is as the starting point for an iterative array definition, e.g.</p>

<pre><code> arr = np.empty((0,3))
 for i in range(10):
     arr = np.append(arr, [i,i+1,i+2])
</code></pre>

<p>which is better writen as</p>

<pre><code> ll = []
 for i in range(10):
     ll.append([i,i+1,i+2])
 arr = np.array(ll)
</code></pre>

<p>or</p>

<pre><code> arr = np.empty((10,3))
 for i in range(10):
     arr[i,:]=[i,i+1,i+2]
</code></pre>

<p>repeated array concatenate is slower.</p>

python numpy