TopolearnModel weights, optimizer slots and counters are variables, and gradients are taken with respect to them.
Because tensors are immutable, TensorFlow keeps changing state in `tf.Variable` objects. A variable wraps a tensor with a fixed dtype and (by default) fixed shape, and lets you update it in place:
v.assign(new_value),v.assign_add(delta),v.assign_sub(delta).- Slices of a variable can be assigned too:
v[0].assign(0.0).
Using a variable in an expression, like v * 2, reads its current value and returns an ordinary tensor.
Variables matter for three reasons:
- Gradients:
tf.GradientTapeautomatically watches trainable variables, so gradients of a loss with respect to weights just work. - Tracking: Keras layers,
tf.Moduleobjects and checkpoints find variables assigned as attributes, which is how saving andtrainable_variableswork. - Placement: a variable lives on one device and is updated there, avoiding copies.
Set trainable=False for state that should not receive gradients, like step counters or batch-norm moving averages.
You rarely create variables by hand in Keras code (layers do it in build), but you will meet them in custom training loops, optimizers and low-level modules.
Example
import tensorflow as tf
w = tf.Variable([1.0, 2.0], name="w")
print(w.numpy()) # [1. 2.]
w.assign([3.0, 4.0])
w.assign_add([1.0, 1.0])
print(w.numpy()) # [4. 5.]
w[0].assign(0.0) # sliced assignment
print(w.numpy()) # [0. 5.]
print((w * 2).numpy()) # [ 0. 10.] reads give plain tensors
step = tf.Variable(0, trainable=False)
step.assign_add(1)
print(step.numpy(), w.trainable, step.trainable) # 1 True FalseRemember
tf.Variable= mutable state with fixed dtype and shape- Update with
assign,assign_add,assign_sub - Trainable variables are watched by GradientTape automatically
Watch out for
- Rebinding
w = w + 1, which replaces the variable with a tensor instead of updating it; usew.assign_add(1).