The most important outage lesson is not that one AI service failed. Every distributed dependency fails eventually. The lesson is that application code must define what happens when the dependency is slow, unavailable, inconsistent, or temporarily restricted.
Python services should use explicit network timeouts. A request without a timeout can occupy a worker until the connection is released, turning a provider incident into your own capacity incident. Retries should use exponential backoff with jitter, and they should be capped so a queue does not grow without limit.
For user-facing features, define graceful degradation. A summarization screen might show the original document and a “try again” action. A support workflow might use a cached answer or route the request for human review. A background job can pause and resume from a checkpoint instead of starting from zero.
Observability completes the design. Track success rate, latency, timeout rate, fallback rate, queue age, and token cost by provider. During an incident, these measurements tell you whether the problem is external, internal, or caused by a particular model or region.
Resilience is not a one-time library choice. It is a product decision about what users are allowed to lose and what the system must preserve. Write that decision down, test it, and make the failure path as intentional as the happy path.