Raise MemoryError when I am fitting a sequence to sequnce LSTM using Keras+Theano
<p>I was trying to implement a sequence to sequence language model. During training process, the model takes in a sequence of 50d word vectors generated by GloVe, and output 1-to-V(V is the size of vocabulary) vector meaning the next word which thus can be regarded as the distribution of next word corresponding to the input word vector at current timestep in test process, and I tried with a 112-word vocabulary.</p>

<p>Then, I built two models as following:</p>

<pre><code>model1 = Sequential()
model1.add(LSTM(112, return_sequences=True, input_shape=(31, 50)))

model2 = Sequential()
model2.add(LSTM(112, return_sequences=True, input_shape=(31, 50)))
model2.add(TimeDistributed(Dense(112, activation="linear")))
</code></pre>

<p>When I tried to fit them by</p>

<pre><code>model.fit(X, Y, batch_size=128, nb_epoch=256, validation_rate=0.1)
</code></pre>

<p>The first model <code>model1</code> crashed and raised MemoryError, but the second model <code>model2</code> normally finished. X has the shape of <code>(number_of_sentences, max_words_in_one_sentence, 50)</code>, and Y has the shape of <code>(number_of_sentences, max_words_in_one_sentence, 112)</code>. In this example, <code>number_of_sentences=10000, max_words_in_one_sentence=13</code>.</p>

<p>I am wondering what happened when I appended a new time-distributed-dense to a LSTM layer, and which one is the model I want to implement my language model.</p>

<p>What happened is that your computing device (probably a GPU) ran out of memory. I suspect it is a NVIDIA card (due to lack of alternatives), so check the output of <code>nvidia-smi</code> to see if you run into memory issues.</p>

<p>Depending on the backend (Theano or TensorFlow) you may experience different behaviours in memory usage, so switching backends may be a solution in some cases.</p>

<p>In case you are using Theano the issue might be the <code>TimeDistributed</code> wrapper. <code>TimeDistributed</code> does this when no batch size is specified:</p>

<pre><code>K.reshape(x, (-1,) + input_shape[2:])
</code></pre>

<p>so it basically reshapes <code>x</code> from <code>(batchsz,timesteps,units)</code> to <code>(batchsz*timesteps,units)</code>. However, reshape is currently allocating a new array if the reshaped one is not C-contiguous (data is sorted as a n-dimensional C array would be), which I suspect it is not in your case.</p>

<p>What you can try is to specify a fixed batch size for your input in which case <code>TimeDistributed</code> will work through the input sequentially using <code>K.rnn</code> without allocation as much memory.</p>

python keras lstm language-model