TopolearnLiteRT (formerly TensorFlow Lite) runs models on billions of phones and microcontrollers.
LiteRT, the new name of TensorFlow Lite, is Google's runtime for on-device inference on Android, iOS, embedded Linux and microcontrollers. Models use the compact .tflite flatbuffer format.
Converting:
- Start from a SavedModel:
tf.lite.TFLiteConverter.from_saved_model(path). For Keras 3 models,model.export(path)first. converter.optimizations = [tf.lite.Optimize.DEFAULT]applies dynamic-range quantisation: weights stored as int8, roughly 4x smaller.- Full integer quantisation also quantises activations: provide a
representative_datasetgenerator yielding sample inputs so the converter can calibrate ranges. Needed for many NPUs and microcontrollers. float16quantisation halves size with minimal accuracy loss on GPU delegates.
Run it with an interpreter: allocate_tensors(), set inputs by index, invoke(), read outputs. In TF 2.20, tf.lite.Interpreter is deprecated in favour of the standalone ai-edge-litert package, which has the same API. On devices, delegates (GPU, NNAPI, Core ML, Hexagon) accelerate supported ops.
Always compare the converted model's outputs and accuracy to the original on real data, especially after quantisation, and check that every op converted (unsupported ops fail conversion unless you allow select TF ops).
Example
import numpy as np
import tensorflow as tf
model = tf.keras.Sequential([tf.keras.Input((4,)),
tf.keras.layers.Dense(8, activation="relu"),
tf.keras.layers.Dense(2)])
model.export("exported") # SavedModel
converter = tf.lite.TFLiteConverter.from_saved_model("exported")
converter.optimizations = [tf.lite.Optimize.DEFAULT] # int8 weights
tflite_bytes = converter.convert()
open("model.tflite", "wb").write(tflite_bytes)
interp = tf.lite.Interpreter(model_content=tflite_bytes) # or ai_edge_litert
interp.allocate_tensors()
inp = interp.get_input_details()[0]
out = interp.get_output_details()[0]
interp.set_tensor(inp["index"], np.random.rand(1, 4).astype(np.float32))
interp.invoke()
print(interp.get_tensor(out["index"]).shape) # (1, 2)Remember
- Convert from a SavedModel;
Optimize.DEFAULTquantises weights - Full int8 needs a representative dataset
- Validate converted outputs against the original model
Watch out for
- Shipping a quantised model without measuring its accuracy on real data.