Retry / Timeout Strategies

Why timeouts and retries belong together

Distributed calls can fail because a dependency is down, slow, overloaded, unreachable, or successful but invisible to the caller because the response was lost. Timeouts bound waiting. Retries attempt recovery. Both need explicit budgets.

Timeout design

A system may have separate connect, read, and overall request timeouts, but the most important concept is the deadline: the original caller has a finite amount of time for the entire operation. Downstream services should not invent a fresh timeout that exceeds the remaining budget.

Suppose an API request has 800 ms remaining after authentication and retrieval. If the model service is given a five-second timeout, it can continue consuming resources long after the user-facing request has already become hopeless. Deadline propagation keeps the call chain aligned.

Retry policy

Retries are appropriate mainly for transient and idempotent failures. A failed GET can often be retried safely. A payment, reservation, or job submission may require an idempotency key because the first attempt might already have succeeded even though the client did not receive the response.

Common retry strategies use exponential backoff with jitter. Exponential backoff increases the delay between attempts; jitter randomizes those delays so thousands of clients do not retry simultaneously and create a thundering herd.

Retry storms

The dangerous case is an overloaded dependency that responds slowly. Clients time out, retry, and thereby send more traffic to the same overloaded service. This raises load, increases latency, triggers more timeouts, and can create a cascading failure.

Production systems therefore cap retries, impose total retry budgets, respect server-provided retry hints, and often combine retries with circuit breaking, rate limits, concurrency bounds, and backpressure.

Example

A gateway gives a model service 1.5 seconds of the user's total 2-second deadline. If a request fails with a transient network error, it may make one retry after jittered backoff. The retry still must fit inside the original deadline. If the service returns a non-retryable validation error, the gateway surfaces it immediately rather than consuming the remaining budget.

The core idea

Timeouts decide how long the system is willing to wait. Retries decide when another attempt is worth its cost. Both are capacity controls, not generic cures for failures.