7. Failure Handling and Resilience

Remote calls fail differently from local calls: they can be slow, time out, partially succeed, or return after the caller has given up. Resilience is not “retry everything.” Good resilience limits the time and resources spent on failing work and prevents a small dependency problem from becoming a system-wide outage.

Evidence: S17, S18, S19

Failure toolbox

Timeout: stop waiting when useful work can no longer finish in time.

Retry: use only for failures that may be transient and only when the operation is safe to repeat.

Exponential backoff + jitter: spread retries over time and reduce synchronized retry storms.

Circuit breaker: stop calling a persistently failing dependency for a period.

Bulkhead/isolation: keep one dependency or workload from exhausting all shared resources.

Graceful degradation: keep the critical path working when optional functionality fails.

Load shedding/rate limiting: reject work before overload destroys the whole service.

Evidence: S17, S18, S19, S3, S45

Checklist

• Every remote call has a finite timeout.

• Retryable and non-retryable errors are distinguished.

• Retries are bounded.

• Retries use backoff; jitter is used when many clients can retry together.

• The operation is idempotent or otherwise protected before automatic retry.

• Retry behavior is not duplicated at many layers without a retry budget.

• Persistent failure has a fail-fast/circuit-breaker strategy where useful.

• Critical resources have concurrency or pool limits.

• Optional dependencies can degrade without taking down the critical path where business allows.

• Overload behavior is intentional: throttle, queue, shed, or reject.

• Failure behavior is tested, not only the success path.

Evidence: S17, S18, S19, S45

Retry decision flow

Figure 5. Retry only transient failures, only when repetition is safe, and always with bounds.

Evidence: S17, S18, S19