Failover protects availability, but it can also turn one logical request into several billable attempts. A timeout can lead to a retry, then a different model, and sometimes another provider. The user sees one answer; the cost ledger may contain three calls. The control objective is therefore not simply “add a fallback.” It is to make the fallback path finite, price-aware, observable, and appropriate for the work.
Last reviewed: 2026-08-12
Direct answer
Set a fallback route cost ceiling before you enable automatic failover. Start by defining the maximum number of attempts for one logical request, the error classes that can trigger each attempt, and the highest acceptable estimated cost for the backup route. Keep the primary and backup model roles explicit. A fallback should be a controlled exception path, not an alternate default that can run indefinitely.
The first distinction is between a retry and a failover. A retry sends the request to the same route again. A failover sends it to a different model or provider. The CometAPI failover guide recommends using failover for provider-side conditions such as timeouts, HTTP 429 rate limits, HTTP 502, 503, or 504 responses, and temporary model unavailability. It advises against rerouting 400, 401, or 403 errors because those usually indicate a malformed request, an authentication problem, or a permission problem that another route will not fix.
Translate that distinction into a per-request budget. If a primary attempt can cost P, a primary retry can cost R, and a fallback attempt can cost F, the worst-case request allowance is P + R + F. Use the current price and token assumptions for the exact CometAPI route in each term. Do not assume that the backup model has the same input, output, context, or service-tier price. The allowance should include both input and output estimates, and it should be checked before an additional attempt is dispatched. When the allowance is exhausted, return a controlled error or a lower-cost degraded response instead of silently adding another model.
An illustrative starting policy looks like this. The values are policy examples, not CometAPI tariffs; tune them with observed usage and your product’s latency and quality requirements.
fallback_policy:
primary_route: primary_text_model
fallback_routes:
- backup_text_model
retry_primary: 1
fallback_attempts: 1
max_attempts_per_request: 3
trigger_statuses: [timeout, 429, 502, 503, 504, model_unavailable]
never_trigger_statuses: [400, 401, 403]
max_cost_per_logical_request_usd: 0.05
require_cost_check_before_each_attempt: true
log_each_attempt: true
abort_when_ceiling_reached: true
Happy path. The request is admitted with a route, a token estimate, and a remaining budget. The primary route returns a valid response inside its timeout. Record one attempt, its actual usage and estimated cost, then close the trace. No fallback event should be inferred merely because a fallback is configured.
Error path. Classify the primary result before retrying. For a timeout or a retryable 5xx or 429, perform only the configured short retry if the remaining allowance covers it. If the retry fails, recalculate the projected cost of the named fallback and dispatch it only when it remains under the ceiling. On fallback success, return the response with a degraded-route flag and retain the complete attempt history. On fallback failure or a budget breach, stop the chain. This makes availability behavior visible to the caller and keeps the cost decision auditable.
Who this is for
This workflow is for FinOps owners who approve AI API budgets, platform engineers who maintain a unified model gateway, and SRE or application teams responsible for latency and error budgets. It is especially useful when a production workload uses a premium primary model and a cheaper or differently priced backup, when several teams share one CometAPI access path, or when a single request can contain long prompts and expensive outputs.
It is not a substitute for fixing malformed requests or access configuration. Teams still integrating a single model can use the same ideas as a preflight checklist: classify errors, estimate a maximum attempt cost, and preserve enough telemetry to explain an invoice. For a broader ledger workflow, see Trace CometAPI cost and usage for token budgets .
Key takeaways
- Treat the logical request, rather than the individual HTTP call, as the budget unit. Count every retry and fallback attempt against it.
- Allow failover only for provider-side conditions. Do not use a second route to mask 400, 401, or 403 defects.
- Order fallback routes from the most suitable approved option to the last-resort option. The LiteLLM fallback documentation describes ordered fallbacks, configurable retries, request timeouts, failure thresholds, and cooldowns; the same controls are useful design requirements around a CometAPI route.
- Compare route-specific input and output estimates before dispatch. A backup can be operationally healthy and still be too expensive for the remaining request allowance.
- Log each attempt, not only the final response. A route-level cost view exposes whether failover is rare protection or a recurring spend multiplier.
- Test the policy in a non-production environment with a real retryable provider error, then inspect the resulting trace. LiteLLM’s current guidance specifically distinguishes real non-production fallback tests from deprecated mock flags for proxy requests.
- Review quality as well as price. A response that is cheaper but incompatible with the task can create a second request, manual rework, or a customer-visible failure.
Use retry evidence for CometAPI cost reviews alongside this policy when you need to separate ordinary retries from cross-route failovers.
Sources checked
- CometAPI: AI API Failover and Fallback Routing defines failover, separates provider-side failures from application-side errors, and lists route-level observability fields including original model, backup model, retry count, final status, and estimated cost.
- LiteLLM: Fallbacks (Provider Failover) documents in-order fallback groups, retry and timeout settings, failure thresholds, cooldowns, and separate context-window and content-policy fallback classes.
- LiteLLM: Spend Tracking documents calculated response cost, model group and provider attribution, request tags, transaction spend logs, and spend reports by user, team, model, and provider.
- Portkey: Fallbacks documents prioritized provider or model targets, status-code-specific triggers, trace filtering for all attempts, nested strategies, and the fact that one request can invoke multiple differently priced LLMs.
These sources describe gateway patterns and accounting evidence. They do not establish a universal CometAPI fallback price or a universal retry limit. Those values must be read from the exact route and plan used by your workload.
Contract details to verify
Before enabling a fallback chain, record the exact model and provider identity for the primary and every backup. Capture the input and output price units, service tier, context assumptions, and any minimum or maximum request size shown in the current CometAPI pricing material. A model label alone is not enough to calculate a ceiling when routes or tiers differ.
Next, verify the trigger contract. Decide whether a timeout starts when the gateway accepts the request, when the upstream connection opens, or when the first token is expected. Map HTTP 429, 502, 503, and 504 separately from 400, 401, and 403. Confirm whether a response that has begun streaming can be retried safely, and whether the first attempt might continue after a fallback begins. For non-idempotent tool or workflow actions, require an application-level idempotency key or disable automatic replay; do not assume a text-generation retry is safe for every side effect.
Set explicit limits for primary retries, fallback attempts, nested chains, and concurrent requests. A sensible first control is one short primary retry and one fallback attempt, but the right number depends on latency and failure data. Add a cooldown or circuit-breaker rule so a visibly unhealthy primary does not cause every request to pay for the same failed attempt. Record who can change these limits and require a review when route pricing, model availability, or quality changes.
Finally, reconcile gateway telemetry with the provider or CometAPI bill. The CometAPI cost and usage article is a useful companion for preserving raw usage evidence. Compare logical request count, attempt count, input tokens, output tokens, fallback rate, and total estimated cost. Investigate any gap before raising the ceiling.
Failure modes
| Failure mode | What happens | Cost control |
|---|---|---|
| Retry storm | A transient 429 or timeout causes repeated retries across many workers. | Set a small per-request retry limit, a request timeout, and a cooldown for an unhealthy route. |
| Expensive backup | The fallback succeeds but has a higher price or output profile than the primary. | Store route-specific prices and reject the attempt when its projected cost exceeds the remaining allowance. |
| Wrong trigger | A malformed request, bad parameter, or permission error is sent to every route. | Exclude 400, 401, and 403 from automatic failover and alert the owning team. |
| Nested chain | A fallback target contains another router or load balancer, so one request invokes more models than expected. | Flatten the approved chain or enforce a global maximum-attempt counter across nested strategies. |
| Duplicate in-flight work | The primary is still running when the fallback starts, producing two responses or two bills. | Cancel or mark the first attempt, and count both attempts even when only one response is returned. |
| Missing attribution | Only the final model is logged, so the extra call disappears from the cost review. | Emit one sanitized record per attempt and join records with a stable request and trace identifier. |
| Stale price map | A cost calculator uses old model or tier prices and approves an unsafe fallback. | Refresh pricing data, retain the source date, and compare calculated cost with the invoice. |
| Incompatible output | The backup model returns a response that cannot satisfy the task, causing rework or another call. | Test quality and format compatibility before placing a model in the fallback list. |
Use a record shape like this for operational telemetry. It contains routing and usage facts, but no credentials or payload content.
{
"request_id": "req_demo_01",
"trace_id": "trace_demo_01",
"attempt": 2,
"route_role": "fallback",
"model": "backup_text_model",
"provider": "provider_b",
"trigger": "timeout",
"status_code": 200,
"retry_count": 1,
"input_tokens": 1200,
"output_tokens": 300,
"estimated_cost_usd": 0.012,
"final": true
}
Do not log prompts, completions, or authentication material merely to prove that a route was used. Keep the minimum fields needed to reconcile the request, diagnose the trigger, and calculate cost. Hash or truncate any customer reference according to your organization’s privacy policy.
FAQ
Is a retry the same as a fallback? No. A retry repeats the same route; a fallback changes the model or provider. Both can be billable attempts, so both belong in the logical-request budget.
Should every error trigger a fallback? No. The CometAPI guidance treats timeouts, 429, 502, 503, 504, and temporary model unavailability as better candidates. It says 400, 401, and 403 usually require a request or configuration fix instead.
How many attempts should I allow? Begin with the smallest chain that meets your availability objective. One primary retry followed by one approved fallback is an easy baseline to measure. Increase it only when failure data shows a benefit that fits the cost ceiling.
Can a fallback lower my bill? It can, but availability is not a savings guarantee. A backup may have a higher input or output price, and a failed primary attempt may already have incurred cost. Calculate the total path, not only the final call.
What should I alert on? Alert on fallback rate, attempts per logical request, cost per fallback, repeated trigger status, and the share of traffic served by each route. A sudden change in any of these can indicate an outage, a quota problem, a routing change, or a stale price assumption.
How do I validate the chain safely? Use a non-production route and induce a retryable provider error. Verify that the configured order is followed, the chain stops at the attempt limit, every attempt is logged, and the final response identifies the route that served it. Then test a 400 or 401 case and confirm that it does not fan out.
Reader next step
Create a one-page fallback register for each workload. List the primary route, approved backups, trigger statuses, timeout, retry count, maximum attempts, route price snapshot, token assumptions, and owner. Add an alert threshold for fallback rate and a date for refreshing the price snapshot. Run one controlled failure test, attach the sanitized attempt records, and compare the calculated path cost with your normal ledger.
For the access path that will carry production traffic, Start with CometAPI . Keep the ceiling in configuration, keep the route identity in every attempt record, and make a budget exception visible before a fallback chain can spend beyond its intended limit.