Questions about hyperparameter tuning in Keras/Tensorflow : <p>I am studying deep learning recently, mainly rely on Andrew Ng's Deep Learning Specialization on Coursera. </p>

<p>And I want to build my own model to classify <code>MNIST</code> with 99% accuracy (simple MLP model, not CNN). So I use <code>KerasClassifier</code> to wrap my model and use <code>GridsearchCV</code> to fine tune the hyperparameters (including hidden layer number, units number, dropout rate, etc.)</p>

<p>However, when I google "fine tuning", the majority of the results are mainly on "transfer learning", which are just tuning the learning rate, output layer number or freeze layer number.</p>

<p>I know these famous models are capable to deal with many problems with just a little changes. But what if I want to build a tiny model from scratch to handle a special question, what are the common/best practice?</p>

<p>So my questions are mainly about the common/best practice of fine tuning model:</p>

<ol>
<li>What is the common/best way to fine tuning? (I have seen people tune hyperparameters manually, or using scikit-learn's <code>RandomizedSearchCV</code>/<code>GridSearchCV</code>, or <code>hyperas</code>)</li>
<li>Should I use k-fold cross validation? (Because it's the default set of <code>GridSearchCV</code>, and it extremely increases the training time but helps little)</li>
<li>Is it enough to solve most problems by slightly modifying the off-the-shelf models? If not, to what direction should I move on?</li>
</ol>

<p>Thanks!</p>
