Neural Network LSTM Keras multiple inputs
<p>I am trying to implement an <a href="https://keras.io/layers/recurrent/#lstm">LSTM with Keras</a>. My inputs and outputs are like in a regression, that is, I have one timeseries y with T observations that I am trying to predict, and I have N (in my case around 20) input vectors (timeseries) of T observations each that I want to use as inputs.</p>

<p>I know that LSTM's in Keras require a <code>3D tensor with shape (nb_samples, timesteps, input_dim)</code> as an input. However, I am not entirely sure how the input should look like in my case, as I have just one sample of T observations for each input, not multiple samples, i.e. I think that would translate to <code>(nb_samples=1, timesteps=T, input_dim=N)</code>. However, is it better to split each of my inputs into samples of length T/M? T is around a few million observations for me, so how long should each sample in that case be, i.e., how would I choose M?</p>

<p>Also, am I right in that this tensor should look something like:</p>

<p><code>[[[a_11, a_12, ..., a_1M], [a_21, a_22, ..., a_2M], ..., [a_N1, a_N2, ..., a_NM]], [[b_11, b_12, ..., b_1M], [b_21, b_22, ..., b_2M], ..., [b_N1, b_N2, ..., b_NM]], ..., [[x_11, x_12, ..., a_1M], [x_21, x_22, ..., x_2M], ..., [x_N1, x_N2, ..., x_NM]]]</code>, where M and N defined as before and x corresponds to the last sample that I would have obtained from splitting as discussed above? </p>

<p>Given a pandas dataframe with T observations in each column, and N columns, one for each input, how can I create such an input to feed to Keras?</p>

<p>In addition, how should I choose the batch size? It seems that Keras only accepts batch sizes that are multiples of <code>nb_samples</code>? Also, when after I have fitted the model, it seems that I can only predict out-of-sample for inputs that are again of a length that is a multiple of the batch size? How does this work in practice?</p>

<p><strong>I would highly appreciate it if someone could show me a full worked example of an LSTM in Keras (and worst case in another python package, or if that's also not possible in another language) for a situation similar to the one I described: i.e. with one output timeseries and multiple input timeseries (just like in a regression), showing how to fit the model in-sample and then predict out-of-sample.</strong></p>

<p>Thanks very much in advance :)!</p>

<p><strong>Tensor shape</strong></p>

<p>You're right that Keras is expecting a 3D tensor for an LSTM neural network, but I think the piece you are missing is that Keras expects that <em>each observation can have multiple dimensions</em>.  </p>

<p>For example, in Keras I have used word vectors to represent documents for natural language processing.  Each word in the document is represented by an n-dimensional numerical vector (so if <code>n = 2</code> the word 'cat' would be represented by something like <code>[0.31, 0.65]</code>).  To represent a single document, the word vectors are lined up in sequence (e.g. 'The cat sat.' = <code>[[0.12, 0.99], [0.31, 0.65], [0.94, 0.04]]</code>).  A document would be a single sample in a Keras LSTM.</p>

<p>This is analogous to your time series observations.  A document is like a time series, and a word is like a single observation in your time series, but in your case it's just that the representation of your observation is just <code>n = 1</code> dimensions.</p>

<p>Because of that, I think your tensor should be something like <code>[[[a1], [a2], ... , [aT]], [[b1], [b2], ..., [bT]], ..., [[x1], [x2], ..., [xT]]]</code>, where <code>x</code> corresponds to <code>nb_samples</code>, <code>timesteps = T</code>, and <code>input_dim = 1</code>, because each of your observations is only one number.</p>

<p><strong>Batch size</strong></p>

<p>Batch size should be set to maximize throughput without exceeding the memory capacity on your machine, per this <a href="http://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network">Cross Validated post</a>. As far as I know your input does not need to be a multiple of your batch size, neither when training the model and making predictions from it.</p>

<p><strong>Examples</strong></p>

<p>If you're looking for sample code, on the <a href="https://github.com/fchollet/keras/tree/master/examples" rel="nofollow">Keras Github</a> there are a number of examples using LSTM and other network types that have sequenced input.</p>

python neural-network keras recurrent-neural-network lstm