Why does my keras LSTM model get stuck in an infinite loop? : <p>I am trying to build a small LSTM that can learn to write code (even if it's garbage code) by training it on existing Python code. I have concatenated a few thousand lines of code together in one file across several hundred files, with each file ending in <code>&lt;eos&gt;</code> to signify "end of sequence".</p>

<p>As an example, my training file looks like:</p>

<pre><code>
setup(name='Keras',
...
      ],
      packages=find_packages())
&lt;eos&gt;
import pyux
...
with open('api.json', 'w') as f:
    json.dump(sign, f)
&lt;eos&gt;
</code></pre>

<p>I am creating tokens from the words with:</p>

<pre><code>file = open(self.textfile, 'r')
filecontents = file.read()
file.close()
filecontents = filecontents.replace("\n\n", "\n")
filecontents = filecontents.replace('\n', ' \n ')
filecontents = filecontents.replace('    ', ' \t ')

text_in_words = [w for w in filecontents.split(' ') if w != '']

self._words = set(text_in_words)
    STEP = 1
    self._codelines = []
    self._next_words = []
    for i in range(0, len(text_in_words) - self.seq_length, STEP):
        self._codelines.append(text_in_words[i: i + self.seq_length])
        self._next_words.append(text_in_words[i + self.seq_length])
</code></pre>

<p>My <code>keras</code> model is:</p>

<pre><code>model = Sequential()
model.add(Embedding(input_dim=len(self._words), output_dim=1024))

model.add(Bidirectional(
    LSTM(128), input_shape=(self.seq_length, len(self._words))))

model.add(Dropout(rate=0.5))
model.add(Dense(len(self._words)))
model.add(Activation('softmax'))

model.compile(loss='sparse_categorical_crossentropy',
              optimizer="adam", metrics=['accuracy'])
</code></pre>

<p>But no matter how much I train it, the model never seems to generate <code>&lt;eos&gt;</code> or even <code>\n</code>. I think it might be because my LSTM size is <code>128</code> and my <code>seq_length</code> is 200, but that doesn't quite make sense? Is there something I'm missing?</p>
