TopolearnTF Serving is a battle-tested, high-throughput server that needs no Python in production.
TensorFlow Serving is a C++ server that loads SavedModels and exposes them over gRPC (port 8500) and REST (port 8501). The easiest way to run it is the tensorflow/serving Docker image.
It expects a versioned layout: models/<name>/<version>/saved_model.pb, where versions are integers. The server watches the directory and by default serves the highest version, loading new versions and unloading old ones without downtime. Rollback is just removing the bad version's folder or pinning a version in a model config file.
Calling it:
- REST:
POST /v1/models/<name>:predictwith{"instances": [...]}(row format) or{"inputs": ...}(columnar). Thesignature_namefield picks a non-default signature. GET /v1/models/<name>reports version status;.../versions/<n>:predicttargets a specific version.- gRPC with the
tensorflow-serving-apipackage is faster for large tensors.
For throughput, enable server-side batching (--enable_batching with a batching parameters file), which groups concurrent requests into larger batches for the GPU.
Before deploying, inspect signatures with saved_model_cli and put preprocessing inside the model so clients send raw features.
Example
# 1. Export a versioned SavedModel (TF Serving loads numbered subfolders)
python -c "
import tensorflow as tf
m = tf.keras.Sequential([tf.keras.Input((4,)), tf.keras.layers.Dense(2)])
m.export('models/clf/1')"
# 2. Check the signature clients must match
saved_model_cli show --dir models/clf/1 --tag_set serve --signature_def serving_default
# 3. Start the server
docker run -d -p 8501:8501 \
-v "$PWD/models/clf:/models/clf" -e MODEL_NAME=clf tensorflow/serving
# 4. Call the REST API
curl -X POST localhost:8501/v1/models/clf:predict \
-d '{"instances": [[1.0, 2.0, 3.0, 4.0]]}'
# {"predictions": [[...two numbers...]]}Remember
- Layout
models/<name>/<version>/; the highest version is served - REST on 8501 (
:predict), gRPC on 8500 - Enable batching for GPU throughput
Watch out for
- Exporting into
models/clf/directly instead of a numbered version folder, so nothing loads.