TopolearnMost models overfit before they underfit; these tools close the gap between training and validation scores.
Overfitting shows up as training loss falling while validation loss rises. Keras offers several remedies, often combined:
- Dropout(rate): randomly zeroes a fraction of activations in training. Typical rates are 0.1 to 0.5.
- Weight penalties:
kernel_regularizer=keras.regularizers.L2(1e-4)on a layer adds a penalty term to the loss. With Adam, preferAdamW(weight_decay=...), which decouples decay from the gradient. - BatchNormalization: normalises activations using batch statistics in training and moving averages at inference. It mainly stabilises training but also regularises a little.
- Data augmentation layers:
RandomFlip,RandomRotation,RandomZoom,RandomContrast,RandomCropcreate varied copies of each image every epoch. - Early stopping with the callback.
All of these behave differently in training and inference. Keras passes a training argument: fit() sets it to True, while predict(), evaluate() and plain model(x) use inference behaviour. Augmentation layers are therefore no-ops at prediction time.
Regularization penalties appear in model.losses and are added to the main loss by fit(). Placing augmentation inside the model means exported models handle raw images; placing it in a tf.data pipeline runs it on the CPU, in parallel with training.
Example
import numpy as np
import keras
from keras import layers, regularizers
model = keras.Sequential([
keras.Input((32, 32, 3)),
layers.RandomFlip("horizontal"), # augmentation: training only
layers.RandomRotation(0.1),
layers.Conv2D(32, 3, padding="same", use_bias=False),
layers.BatchNormalization(),
layers.Activation("relu"),
layers.GlobalAveragePooling2D(),
layers.Dropout(0.3),
layers.Dense(10, kernel_regularizer=regularizers.L2(1e-4)),
])
x = np.random.rand(4, 32, 32, 3).astype("float32")
a = keras.ops.convert_to_numpy(model(x, training=False))
b = keras.ops.convert_to_numpy(model(x, training=False))
print(np.allclose(a, b)) # True: inference is deterministic
print(len(model.losses)) # 1 (the L2 penalty, added to the loss by fit)Remember
- Dropout, weight decay, BatchNorm and augmentation combat overfitting
training=Truein fit, inference behaviour in predict and model(x)- Prefer AdamW's weight_decay over L2 penalties with Adam
Watch out for
- Calling
model(x, training=True)for inference, which applies dropout and random augmentation.