TopolearnReal inputs like sentences and event lists have different lengths; TF can handle them inside the graph and the served model.
TensorFlow tensors can hold strings (tf.string, byte strings), and tf.strings provides vectorised ops: lower, split, regex_replace, substr, to_number, unicode_decode. Because they are graph ops, preprocessing can live inside a SavedModel and run identically in training and serving.
Variable-length data does not fit a rectangular tensor. A `tf.RaggedTensor` stores rows of different lengths efficiently (values plus row boundaries). tf.strings.split returns one; you can also build them with tf.ragged.constant. Many ops support ragged inputs directly (reduce_sum, concat, map_flat_values), and .to_tensor(default_value) pads to a dense tensor when a layer needs it. row_lengths() gives each row's length.
For sparse data with mostly zeros (huge one-hot features), tf.sparse.SparseTensor stores only indices and values.
For text models, the `TextVectorization` layer standardises, splits and maps words to integer ids. Call adapt(dataset) to build the vocabulary; index 0 is padding and 1 is the out-of-vocabulary token. Use it inside a tf.data map for speed, or inside the model so raw strings can be served.
Example
import tensorflow as tf
words = tf.strings.split(tf.constant(["The cat sat", "hi"]))
print(words) # <tf.RaggedTensor [[b'The', b'cat', b'sat'], [b'hi']]>
print(words.row_lengths().numpy()) # [3 1]
print(words.to_tensor(default_value="").shape) # (2, 3)
print(tf.strings.lower("Hello").numpy()) # b'hello'
r = tf.ragged.constant([[1, 2, 3], [4]])
print(tf.reduce_sum(r, axis=1).numpy()) # [6 4]
vec = tf.keras.layers.TextVectorization(max_tokens=100, output_sequence_length=4)
vec.adapt(["the cat sat", "the dog ran"])
print(vec(["the cat ran"]).numpy())
# [[2 x y 0]]: 'the' is id 2 (most frequent), 0 = padding, 1 = unknown wordsRemember
tf.stringsops keep text preprocessing inside the graph- RaggedTensor holds rows of different lengths;
.to_tensor()pads - TextVectorization:
adaptbuilds the vocabulary; 0 = pad, 1 = OOV
Watch out for
- Doing text preprocessing in Python for training but not in serving, so the two drift apart.