TopolearnWell-tuned retries hide transient failures; badly tuned ones cause outages.
gRPC distinguishes two kinds of retry:
- Transparent retries: when the library knows the request never reached the server application (e.g. the stream was refused), it retries automatically. Always safe.
- Configured retries: a retry policy in the service config for specific methods:
maxAttempts,initialBackoff,maxBackoff,backoffMultiplierandretryableStatusCodes(usually justUNAVAILABLE). Only retry idempotent methods.
Hedging is the aggressive cousin: send the same request to several backends after hedgingDelay without waiting for failure, and use the first response. It cuts tail latency for idempotent reads at the cost of extra load. A method can have a retry policy or a hedging policy, not both.
Safety valves:
- Retry throttling (
retryThrottlingwithmaxTokensandtokenRatio) stops retries when failure rates rise, preventing retry storms. - The call deadline bounds all attempts together.
- Servers can push back with
grpc-retry-pushback-msmetadata.
The service config can come from the client (grpc.service_config channel option) or from the name resolver (DNS TXT records or xDS), letting operators tune policies without redeploying clients. It also sets per-method timeouts (timeout), waitForReady and load-balancing policy.
Example
// Service config JSON, passed as the grpc.service_config channel option
const serviceConfig = {
loadBalancingConfig: [{ round_robin: {} }],
methodConfig: [
{
name: [{ service: "hello.v1.Greeter", method: "SayHello" }],
timeout: "0.5s", // default deadline per call
retryPolicy: {
maxAttempts: 4,
initialBackoff: "0.1s",
maxBackoff: "1s",
backoffMultiplier: 2,
retryableStatusCodes: ["UNAVAILABLE"],
},
},
],
retryThrottling: { maxTokens: 10, tokenRatio: 0.1 },
};
const options = { "grpc.service_config": JSON.stringify(serviceConfig) };
console.log(Object.keys(options)); // [ 'grpc.service_config' ]
// new hello.v1.Greeter(target, credentials, options)Remember
- Transparent retries are automatic and always safe
- Retry policies belong in service config, for idempotent methods only
- Hedging trades extra load for lower tail latency
- Retry throttling prevents retry storms
Watch out for
- Retrying non-idempotent methods like
ChargeCardon UNAVAILABLE, causing duplicate charges.