Neural Network LSTM Keras multiple inputs
<p>I am trying to implement an <a href="https://keras.io/layers/recurrent/#lstm">LSTM with Keras</a>. My inputs and outputs are like in a regression, that is, I have one timeseries y with T observations that I am trying to predict, and I have N (in my case around 20) input vectors (timeseries) of T observations each that I want to use as inputs.</p>

<p>I know that LSTM's in Keras require a <code>3D tensor with shape (nb_samples, timesteps, input_dim)</code> as an input. However, I am not entirely sure how the input should look like in my case, as I have just one sample of T observations for each input, not multiple samples, i.e. I think that would translate to <code>(nb_samples=1, timesteps=T, input_dim=N)</code>. However, is it better to split each of my inputs into samples of length T/M? T is around a few million observations for me, so how long should each sample in that case be, i.e., how would I choose M?</p>

<p>Also, am I right in that this tensor should look something like:</p>

<p><code>[[[a_11, a_12, ..., a_1M], [a_21, a_22, ..., a_2M], ..., [a_N1, a_N2, ..., a_NM]], [[b_11, b_12, ..., b_1M], [b_21, b_22, ..., b_2M], ..., [b_N1, b_N2, ..., b_NM]], ..., [[x_11, x_12, ..., a_1M], [x_21, x_22, ..., x_2M], ..., [x_N1, x_N2, ..., x_NM]]]</code>, where M and N defined as before and x corresponds to the last sample that I would have obtained from splitting as discussed above? </p>

<p>Given a pandas dataframe with T observations in each column, and N columns, one for each input, how can I create such an input to feed to Keras?</p>

<p>In addition, how should I choose the batch size? It seems that Keras only accepts batch sizes that are multiples of <code>nb_samples</code>? Also, when after I have fitted the model, it seems that I can only predict out-of-sample for inputs that are again of a length that is a multiple of the batch size? How does this work in practice?</p>

<p><strong>I would highly appreciate it if someone could show me a full worked example of an LSTM in Keras (and worst case in another python package, or if that's also not possible in another language) for a situation similar to the one I described: i.e. with one output timeseries and multiple input timeseries (just like in a regression), showing how to fit the model in-sample and then predict out-of-sample.</strong></p>

<p>Thanks very much in advance :)!</p>

<p>Below is an example that sets up time series data to train an LSTM.  The model output is nonsense as I only set it up to demonstrate how to build the model.</p>

<pre><code>import pandas as pd
import numpy as np
# Get some time series data
df = pd.read_csv("https://raw.githubusercontent.com/plotly/datasets/master/timeseries.csv")
df.head()
</code></pre>

<p>Time series dataframe:</p>

<pre><code>Date      A       B       C      D      E      F      G
0   2008-03-18  24.68  164.93  114.73  26.27  19.21  28.87  63.44
1   2008-03-19  24.18  164.89  114.75  26.22  19.07  27.76  59.98
2   2008-03-20  23.99  164.63  115.04  25.78  19.01  27.04  59.61
3   2008-03-25  24.14  163.92  114.85  27.41  19.61  27.84  59.41
4   2008-03-26  24.44  163.45  114.84  26.86  19.53  28.02  60.09
</code></pre>

<p>You can build put inputs into a vector and then use pandas <code>.cumsum()</code> function to build the sequence for the time series:</p>

<pre><code># Put your inputs into a single list
df['single_input_vector'] = df[input_cols].apply(tuple, axis=1).apply(list)
# Double-encapsulate list so that you can sum it in the next step and keep time steps as separate elements
df['single_input_vector'] = df.single_input_vector.apply(lambda x: [list(x)])
# Use .cumsum() to include previous row vectors in the current row list of vectors
df['cumulative_input_vectors'] = df.single_input_vector.cumsum()
</code></pre>

<p>The output can be set up in a similar way, but it will be a single vector instead of a sequence:</p>

<pre><code># If your output is multi-dimensional, you need to capture those dimensions in one object
# If your output is a single dimension, this step may be unnecessary
df['output_vector'] = df[output_cols].apply(tuple, axis=1).apply(list)
</code></pre>

<p>The input sequences have to be the same length to run them through the model, so you need to pad them to be the max length of your cumulative vectors:</p>

<pre><code># Pad your sequences so they are the same length
from keras.preprocessing.sequence import pad_sequences

max_sequence_length = df.cumulative_input_vectors.apply(len).max()
# Save it as a list   
padded_sequences = pad_sequences(df.cumulative_input_vectors.tolist(), max_sequence_length).tolist()
df['padded_input_vectors'] = pd.Series(padded_sequences).apply(np.asarray)
</code></pre>

<p>Training data can be pulled from the dataframe and put into numpy arrays.  <strong>Note that the input data that comes out of the dataframe will not make a 3D array.  It makes an array of arrays, which is not the same thing.</strong></p>

<p>You can use hstack and reshape to build a 3D input array.</p>

<pre><code># Extract your training data
X_train_init = np.asarray(df.padded_input_vectors)
# Use hstack to and reshape to make the inputs a 3d vector
X_train = np.hstack(X_train_init).reshape(len(df),max_sequence_length,len(input_cols))
y_train = np.hstack(np.asarray(df.output_vector)).reshape(len(df),len(output_cols))
</code></pre>

<p>To prove it:</p>

<pre><code>&gt;&gt;&gt; print(X_train_init.shape)
(11,)
&gt;&gt;&gt; print(X_train.shape)
(11, 11, 6)
&gt;&gt;&gt; print(X_train == X_train_init)
False
</code></pre>

<p>Once you have training data you can define the dimensions of your input layer and output layers.</p>

<pre><code># Get your input dimensions
# Input length is the length for one input sequence (i.e. the number of rows for your sample)
# Input dim is the number of dimensions in one input vector (i.e. number of input columns)
input_length = X_train.shape[1]
input_dim = X_train.shape[2]
# Output dimensions is the shape of a single output vector
# In this case it's just 1, but it could be more
output_dim = len(y_train[0])
</code></pre>

<p>Build the model:</p>

<pre><code>from keras.models import Model, Sequential
from keras.layers import LSTM, Dense

# Build the model
model = Sequential()

# I arbitrarily picked the output dimensions as 4
model.add(LSTM(4, input_dim = input_dim, input_length = input_length))
# The max output value is &gt; 1 so relu is used as final activation.
model.add(Dense(output_dim, activation='relu'))

model.compile(loss='mean_squared_error',
              optimizer='sgd',
              metrics=['accuracy'])
</code></pre>

<p>Finally you can train the model and save the training log as history:</p>

<pre><code># Set batch_size to 7 to show that it doesn't have to be a factor or multiple of your sample size
history = model.fit(X_train, y_train,
              batch_size=7, nb_epoch=3,
              verbose = 1)
</code></pre>

<p>Output:</p>

<pre><code>Epoch 1/3
11/11 [==============================] - 0s - loss: 3498.5756 - acc: 0.0000e+00     
Epoch 2/3
11/11 [==============================] - 0s - loss: 3498.5755 - acc: 0.0000e+00     
Epoch 3/3
11/11 [==============================] - 0s - loss: 3498.5757 - acc: 0.0000e+00 
</code></pre>

<p>That's it.  Use <code>model.predict(X)</code> where <code>X</code> is the same format (other than the number of samples) as <code>X_train</code> in order to make predictions from the model.</p>

python neural-network keras recurrent-neural-network lstm