Padding time-series subsequences for LSTM-RNN training : <p>I have a dataset of time series that I use as input to an LSTM-RNN for action anticipation. The time series comprises a time of 5 seconds at 30 fps (i.e. 150 data points), and the data represents the position/movement of facial features.</p>

<p>I sample additional sub-sequences of smaller length from my dataset in order to add redundancy in the dataset and reduce overfitting. In this case I know the starting and ending frame of the sub-sequences.</p>

<p>In order to train the model in batches, all time series need to have the same length, and according to many papers in the literature padding should not affect the performance of the network.</p>

<p>Example:</p>

<p>Original sequence:</p>

<pre><code> 1 2 3 4 5 6 7 8 9 10
</code></pre>

<p>Subsequences:</p>

<pre><code>4 5 6 7
8 9 10
2 3 4 5 6
</code></pre>

<p>considering that my network is trying to <em>anticipate</em> an action (meaning that as soon as P(action) > threshold as it goes from t = 0 to T = tmax, it will predict that action) will it matter where the padding goes? </p>

<p><strong>Option 1</strong>: Zeros go to substitute original values</p>

<pre><code>0 0 0 4 5 6 7 0 0 0
0 0 0 0 0 0 0 8 9 10
0 2 3 4 5 6 0 0 0 0
</code></pre>

<p><strong>Option 2</strong>: all zeros at the end</p>

<pre><code>4 5 6 7 0 0 0 0 0 0 
8 9 10 0 0 0 0 0 0 0
2 3 4 5 0 0 0 0 0 0
</code></pre>

<p>Moreover, some of the time series are missing a number of frames, but it is not known which ones they are - meaning that if we only have 60 frames, we don't know whether they are taken from 0 to 2 seconds, from 1 to 3s, etc. These need to be padded before the subsequences are even taken. What is the best practice for padding in this case?</p>

<p>Thank you in advance.</p>
