Is it normal that transfer learning (VGG16) performs worse on CIFAR-10? : <p><strong>Note:</strong> I am not sure this is the right website to ask these kind of questions. Please tell me where I should ask them before downvoting this "because this isn't the right place to ask". Thanks!</p>

<p>I am currently experimenting with deep learning using <a href="https://keras.io" rel="nofollow noreferrer">Keras</a>. I tried already a model similar to the one to be found on <a href="https://keras.io/examples/cifar10_cnn/" rel="nofollow noreferrer">the Keras example</a>. This yields expecting results:</p>

<ul>
<li>80% after 10-15 epochs without data augmentation before overfitting around the 15th epoch and</li>
<li>80% after 50 epochs with data augmentation without any signs of overfitting.</li>
</ul>

<p>After this I wanted to try transfer learning. I did this by using the <a href="https://keras.io/applications/#vgg16" rel="nofollow noreferrer">VGG16</a> network without retraining its weights (see code below). This gave very poor results: 63% accuracy after 10 epochs with a very shallow curve (see picture below) which seems to be indicating that it will achieve acceptable results only (if ever) after a very long training time (I would expect 200-300 epochs before it reaches 80%).</p>

<p><a href="https://i.stack.imgur.com/MWmsT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MWmsT.png" alt="Transfer learning results"></a></p>

<p>Is this normal behavior for this kind of application? Here are a few things I could imagine to be the cause of these bad results:</p>

<ul>
<li>the CIFAR-10 dataset has images of <code>32x32</code> pixels, which might be too few for the VGG16 net</li>
<li>The filters of VGG16 are not good for CIFAR-10, which would be solved by setting the weights to <code>trainable</code> or by starting with random weights (only copying the model and not the weights)</li>
</ul>

<p>Thanks in advance!</p>

<hr>

<p><strong>My code:</strong></p>

<p><em>Note that the inputs are 2 datasets (50000 training images and 10000 testing images) which are labeled images with shape <code>32x32x3</code>. Each pixel value is a float in the range <code>[0.0, 1.0]</code>.</em></p>

<pre class="lang-py prettyprint-override"><code>import keras

# load and preprocess data...

# get VGG16 base model and define new input shape
vgg16 = keras.applications.vgg16.VGG16(input_shape=(32, 32, 3),
                                       weights='imagenet',
                                       include_top=False)

# add new dense layers at the top
x = keras.layers.Flatten()(vgg16.output)
x = keras.layers.Dense(1024, activation='relu')(x)
x = keras.layers.Dropout(0.5)(x)
x = keras.layers.Dense(128, activation='relu')(x)
predictions = keras.layers.Dense(10, activation='softmax')(x)

# define and compile model
model = keras.Model(inputs=vgg16.inputs, outputs=predictions)
for layer in vgg16.layers:
    layer.trainable = False
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# training and validation
model.fit(x_train, y_train,
          batch_size=256,
          epochs=10,
          validation_data=(x_test, y_test))

model.evaluate(x_test, y_test)
</code></pre>
