How does the AI service remain available during model or server failures?

Failure Domains Instead of Duplicate Processes

The service is viewed in layers: gateway, inference, vector and metadata storage, identity provider, and source systems. Each layer has a defined failure behavior. At least 2 inference instances increase availability only if they do not share the same single power supply, GPU, machine, or other common failure cause. A load balancer only sends new requests to instances whose readiness check is successful.

Health checks must detect more than one running process. For example, Qdrant provides /healthz, /livez, and /readyz; the serving system should provide comparable liveness and readiness signals. Readiness should only be positive when the model and necessary resources are loaded. Otherwise, a freshly started but not yet usable process receives production traffic. Time limits, limited retries, and circuit breakers prevent failure chains.

State data is replicated according to the RPO and additionally backed up. Qdrant points out that the default replication factor of 1 does not provide an additional copy; a factor of 2 doubles the relevant data copies and thus the storage requirement. Replication, however, does not protect against logical deletion. Backup and restore testing remain necessary.

During a model change, old and new versions run in parallel. A canary receives a small, controlled portion of the load; quality, error, and latency metrics determine whether to scale up or rollback. A technical fallback can be a smaller tested model, a pure search, or a clear unavailability message. It must not generate a seemingly complete response with unverified quality. Kubernetes PodDisruptionBudgets limit voluntary interruptions but, according to documentation, do not protect against all involuntary failures. Therefore, regular drills for instance, node, model, and data storage failures are conducted.