5 Tensorflow Callbacks for Quick and Easy Training

If you’re new to the world of machine learning, neural networks and deep learning, or just want to get up and running with your own AI/ML model as quickly as possible, then Tensorflow callbacks are an excellent way to do so. While there are many options out there for using Tensorflow that can be useful in specific situations, if you want something quick and dirty, callbacks are the way to go. Here’s how they work.

What is Tensorflow Callback?

Tensorflow Callbacks are pieces of code you can set up in order to monitor certain aspects of a model’s training process. For example, you could have a callback that sends an email every time your training error exceeds some threshold value or one that takes new data (input) and adjusts your model based on it. There are lots of different callbacks, which we’ll get into later in this post, but first, let’s look at how they work.

EarlyStopping

We use early stopping in training, which stops our model from overfitting data. We can do that by keeping track of our validation accuracy on each step of training. If it gets better, we stop training earlier than expected. This helps us avoid overfitting issues because it stops our model from learning patterns on data and continuing forever even though those patterns are irrelevant in real life predictions. stop training when it measures no progress on the validation set after a number of epochs

tf.keras.callbacks.EarlyStopping(
    monitor="val_loss",
    min_delta=0,
    patience=0,
    verbose=0,
    mode="auto",
    baseline=None,
    restore_best_weights=False,
)

earlystop= keras.callbacks.EarlyStopping(patience=5)

history = model.fit(X_train, y_train, epochs=1000, callbacks=[earlystop], validation_split = 0.2)

ModelCheckpoint

The first step to using callbacks is creating a model that uses them. Luckily, Keras makes it easy with its ModelCheckpoint class. Using model checkpoints is as simple as adding keras.callbacks.ModelCheckpoint() in your training loop’s call to fit(). When using callbacks, it’s recommended that you set your own saving interval (the number of batches between saving). There are a few methods available that do so

tf.keras.callbacks.ModelCheckpoint(
    filepath,
    monitor="val_loss",
    verbose=0,
    save_best_only=False,
    save_weights_only=False,
    mode="auto",
    save_freq="epoch",
    options=None,
    initial_value_threshold=None,
    **kwargs
)

# Save Check point
checkpoint_cb = keras.callbacks.ModelCheckpoint("testcheckpoint.h5",save_best_only=True)

# Early Stopping
earlystop= keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)

history = model.fit(X_train, y_train, epochs=1000, callbacks=[checkpoint_cb, earlystop], validation_split = 0.2)

TerminateOnNaN

One of TensorFlow’s greatest strengths is that it has a wide range of inputs you can feed into it. This means, however, that you must anticipate any edge cases and account for them in your code. The TerminateOnNaN function allows you to specify how TensorFlow should handle values that are not numbers (i.e., NaN).

# Callback that terminates training when a NaN loss is encountered.
nan = tf.keras.callbacks.TerminateOnNaN()

history = model.fit(X_train, y_train, epochs=1000, callbacks=[nan], validation_split = 0.2)

LearningRateScheduler

The LearningRateScheduler is a nifty class that helps manage learning rate training. The LearningRateScheduler automatically scales learning rates over time, resulting in steady improvements that are not interrupted by sudden ups and downs, like what can happen when you manually adjust your learning rate from trial to trial. Using a LearningRateScheduler can save you hours of work as you train on custom datasets or complex models. The best part? It’s extremely easy to use!

 def scheduler(epoch, lr):
   if epoch < 10:
     return lr
   else:
     return lr * tf.math.exp(-0.1)

callback = tf.keras.callbacks.LearningRateScheduler(scheduler)

history = model.fit(X_train, y_train, epochs=1000, callbacks=[callback], validation_split = 0.2)

ReduceLROnPlateau

During training, many model parameters will fluctuate around their correct values. At some point, these variations slow or stop; then, there is no longer any point in continuing training—the parameter estimates have reached a plateau. The ReduceLROnPlateau function looks at your model’s history of updates and determines whether it has stopped improving. It returns True if there are no more updates that improve your metrics or False otherwise. In practice, you rarely want to train until you reach a plateau.

tf.keras.callbacks.ReduceLROnPlateau(
    monitor="val_loss",
    factor=0.1,
    patience=10,
    verbose=0,
    mode="auto",
    min_delta=0.0001,
    cooldown=0,
    min_lr=0,
    **kwargs
)

reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
                              patience=5, min_lr=0.001)
model.fit(X_train, Y_train, callbacks=[reduce_lr])

Arguments

• monitor: quantity to be monitored.
• factor: factor by which the learning rate will be reduced. new_lr = lr * factor.
• patience: number of epochs with no improvement after which learning rate will be reduced.
• verbose: int. 0: quiet, 1: update messages.
• min_lr: lower bound on the learning rate.