TopolearnWatching loss curves, learning rates and weight distributions across runs is how you debug and tune training.
TensorBoard is TensorFlow's visualisation tool. Training code writes event files to a log directory, and tensorboard --logdir logs serves a web UI that plots them, comparing every run found under the directory.
With Keras, add the callback:
tf.keras.callbacks.TensorBoard(log_dir="logs/run1", histogram_freq=1)It logs losses and metrics per epoch, weight histograms, and can profile a range of batches with profile_batch=(10, 20).
In custom loops, use the tf.summary API:
writer = tf.summary.create_file_writer("logs/run1")- inside
with writer.as_default():, calltf.summary.scalar(name, value, step=step),tf.summary.histogram,tf.summary.imageortf.summary.text.
Always pass step, which becomes the x-axis. Summaries work inside tf.function, so you can log from the compiled train step.
Give each experiment its own subdirectory (for example including a timestamp or hyperparameters in the name) so runs can be overlaid. TensorBoard also includes the Profiler plugin for step-time breakdowns, and the projector for embeddings. Other frameworks, including PyTorch, can write TensorBoard logs too.
Example
import tensorflow as tf
writer = tf.summary.create_file_writer("logs/run1")
w = tf.Variable(5.0)
opt = tf.keras.optimizers.SGD(learning_rate=0.1)
for step in range(100):
with tf.GradientTape() as tape:
loss = (w - 2.0) ** 2
opt.apply_gradients([(tape.gradient(loss, w), w)])
with writer.as_default():
tf.summary.scalar("loss", loss, step=step)
tf.summary.scalar("w", w, step=step)
writer.flush()
# With fit():
# model.fit(x, y, callbacks=[tf.keras.callbacks.TensorBoard("logs/run2")])
# Then in a shell: tensorboard --logdir logsRemember
- Write event files, view with
tensorboard --logdir - Keras: the TensorBoard callback; custom loops:
tf.summary.*with a step - One subdirectory per run makes comparisons easy
Watch out for
- Logging every run to the same directory, which tangles curves from different experiments.