Why is the network not learning as effective (Tensorflow - LSTMs for text generation)?
<p>So my query is this: when I initially ran a 2 LSTM network (each 512 units) on the Shakespeare corpus, I got a pretty decent output after 2 epochs of training (where each epoch = one cycle through the dataset). 
Here it is: </p>

<p><em>best ward of mine</em>
<em>him that you a man of make the pein to the noble from his day?</em></p>

<p><em>DUKE VINGEN:
So that man be, mure the this more to the part that the tongue were an one shall you an every wall.</em></p>

<p><em>HALLES:</em>
<em>Why, your hall thou to the good word to my sole mister of the men and the twouse to your ever and me the appice to be a forting would the crease.</em></p>

<p><em>HARLET:</em>
<em>I would not the ching and not the good souls as be be he of the seement that is dead mone stains to me the ufber me do and with the sures and to conforn the tre</em></p>

<p>It was still learning, with loss of 1.73 and the graph of loss was not levelling off. However, in all subsequent runs (10) it has not been able to achieve anywhere as near as good results - after 6 epochs, for example, in my best run subsequently it had a loss of 2.43 and was levelling off - 
this is the output:</p>

<p><em>friar?
How doth my l mo toee  or inrou
 o  inain  me c ahes oisteese an th thet  ot spr  h 
 sat t heooi to  yerton dhe  tha   dou tr hum es teride t an  Mose lhre moud wilh the will  add toe  exereaasne op ther sa e  and ho m reevion 
hatd tik  t e thane ofore te,t
hhe serined   o chane the   ertrs  aine thele aelt  To tee  g te
 ou so yos we tiie be rere th livt hir but at jnt hu
As   ehmug te to fare t ce
To tha mhn hi did r t ter  oe hir  rat ft thr ionyoee wh  eoo that e cade tientta To
tike  a r  hee  ta t he</em></p>

<p>This is similar to the outputs after 4, 5, 6 epochs. Most commonly, in the runs subsequent to the initial run, the network would level off learning after a loss of 2.70.</p>

<p><a href="http://i.stack.imgur.com/BoUzy.png" rel="nofollow">This is the graph of the loss</a>
 and I have also posted the code:</p>

<pre><code>from __future__ import absolute_import, division, print_function
import os
import numpy as np
import tflearn
from tflearn.data_utils import *
from tflearn.layers.estimator import regression


inputs, targets, char_dict = \
    textfile_to_semi_redundant_sequences("shakespeare_input.txt", seq_maxlen=20) #helper - vectorises text



LSTM = tflearn.input_data([None, 20, len(char_dict)])
LSTM = tflearn.lstm(LSTM, 512, return_seq=True, restore=True, dropout = 0.5)
LSTM = tflearn.lstm(LSTM, 512, restore=True, dropout = 0.5)
LSTM = tflearn.fully_connected(LSTM, len(char_dict), activation='softmax')

LSTM = tflearn.regression(LSTM, optimizer= 'adam', loss='categorical_crossentropy',
                       learning_rate=0.001) 

LSTMmodel = tflearn.SequenceGenerator(LSTM, dictionary=char_dict,
                              seq_maxlen=20,
                              clip_gradients=5.0, tensorboard_verbose=0,tensorboard_dir='pathfile/logs')

#LSTMmodel.load('/pathfile/LSTMmodel.tfl')
for i in range(10):
    print("-- TESTING...")
    starting = random_sequence_from_textfile("shakespeare_input.txt", 20)
    output_path = 'pathfile/epoch_' + str(i) + '_output.txt' 
    generated_output = LSTMmodel.generate(500, temperature=1.0, seq_seed=starting)
    text_file = open(output_path, "w") #save the outputs to a text file - allows us to view progress of model
    text_file.write("With temperature 1.0: \n \n \n") #two different temperatures - higher temp = more novel
    text_file.write(generated_output)
    generated_output = LSTMmodel.generate(500, temperature=0.5, seq_seed=starting) #lower temp = more accurate to original text
    text_file.write("\n \n \n With temperature 0.5: \n \n \n")
    text_file.write(generated_output)
    text_file.close()
    print("-- TRAINING...")
    LSTMmodel.fit(inputs, targets, batch_size=200, n_epoch=1, run_id='Shakespeare_Generator',shuffle = True)
    print("-- SAVING MODEL...")
    if (i%2==0):
        LSTMmodel.save("pathfile/LSTMmodel.tfl")
    else:
        LSTMmodel.save("pathfile//LSTMmodel2.tfl")
    print("-- EPOCH " + str(i+1) +" COMPLETE...")
</code></pre>

<p>Since I am running this on an old dual core i3 with 4gb of RAM, the network takes 6 hr 45 mins to go through one epoch. And of course, it will take at least two epochs to see results. So unfortunately, I cannot keep tweaking the code and running and re-running. I am also limited by the memory of my computer, as 1gb of the 4 is allocated to Ubuntu - the other 3 allocated to Windows. Therefore, I can only train a small network.</p>

<p>I would really appreciate if someone could provide an answer that is tested and known to work. Sorry if this sounds like an unreasonable request - I do not wish to come across as demanding. </p>

<p>I don't have an answer for this model, but have you tried starting with an existing example for a Shakespeare-generating LSTM, like this one?</p>

<p><a href="https://github.com/sherjilozair/char-rnn-tensorflow" rel="nofollow">https://github.com/sherjilozair/char-rnn-tensorflow</a></p>

<p>It should be a bit quicker to train, and if you're starting from a working example it might be easier to debug where it's going wrong.</p>

python tensorflow deep-learning lstm