Multivariate Regression Neural Network Loss Function : <p>I am doing multivariate regression with a fully connected multilayer neural network in Tensorflow. The network predicts 2 continuous float variables <code>(y1,y2)</code> given an input vector <code>(x1,x2,...xN)</code>, i.e. the network has 2 output nodes. With 2 outputs the network does not seem to converge. My loss function is essentially the L2 distance between the prediction and truth vectors (each contains 2 scalars): </p>

<pre><code>loss = tf.nn.l2_loss(tf.sub(prediction, truthValues_placeholder)) + L2regularizationLoss 
</code></pre>

<p>I am using L2 regularization, dropout regularization, and my activation functions are tanh.</p>

<p><strong>My questions</strong>: Is L2 distance the proper way to calculate loss for a multivariate network output? Are there some tricks needed to get multivariate regression networks to converge (as opposed to single-variable networks and classifiers)?</p>
