Keras - input shape of large 2-dimensional Array
<p>I want to build an array which contains a very large number of elements
number of sequences (batch size) * size of dictionary (unique words in file)
474683 * 22995</p>

<p>each sequence will have some number X of bits turned on which represents a word in the dictionary</p>

<p>the sentence is: "I am the best king"
lets say the dictionary is:</p>

<p>[I, am, king, the, best, animal, toast, ...]</p>

<p>the sequence will look like:</p>

<p>[1,1,1,1,1,0,0,...]</p>

<p>I try to import this to keras and I get an error:</p>

<p>Exception: Error when checking model input: expected lstm_input_9 to have 3 dimensions, but got array with shape (93371, 22995)</p>

<p>From <a href="http://stackoverflow.com/questions/38105999/dimension-mismatch-in-lstm-keras">here</a> we can see that keras expects:
(batch_size, sequence_length, input_dimension)</p>

<p>What can I do about this?</p>

<p>If I try to build an nunpy array which has a sequence length, say 20, I will get a memory error (its something like 26gb), should I just split the array and train on each one separately?</p>

<p>Okay, I decided to split the array and train on each of the split instead of pushing everything to memory:</p>

<pre><code>data_cut = 3

X = np.zeros((len(inputs)/data_cut, max_len, len(words)), dtype=np.bool)
y = np.zeros((len(inputs)/data_cut, len(words)), dtype=np.bool)


# set the appropriate indices to 1 in each one-hot vector
ins = []
for cuts in range(0,data_cut):#number of cuts
    cutLocation = len(inputs)/data_cut*cuts#location of cut
    start = (cutLocation)
    end = (len(inputs)/data_cut) * (cuts + 1)
    ins.append(inputs[start:end])#first half
    for i, example in enumerate(ins[cuts]):
        for t, word in enumerate(example):
            X[i, t, word_labels[word]] = 1
        y[i, word_labels[outputs[i]]] = 1
    model.fit(X, y, batch_size=64, nb_epoch=epochs)
</code></pre>

<p>Of course you want to split then train over the number of splits.</p>

<p>If anyone want to take a stab at making this more efficient please write below.</p>

python arrays keras