Cap AI API Candidate Counts Before Completions Multiply Spend

Last reviewed: 2026-08-13

Direct answer

AI API candidate count cost controls should make one candidate the default, enforce that default before a request reaches the model, and treat every higher count as an explicit budget exception. A request counter is not enough: the cost ledger must settle against aggregate usage returned for the whole request, including output generated for candidates the application later discards.

This matters because candidate count and output length are separate multipliers. The current OpenAI chat-completions contract describes n as the number of choices generated for each input and says generated tokens across all choices are charged. It explicitly recommends keeping n at 1 to minimize cost. A sensible planning reservation is therefore:

reserved output = approved candidate count x per-candidate output ceiling

Then add estimated input usage to obtain a request reservation. Do not present that calculation as the final bill: route pricing, parameter support, output-cap semantics, and actual generation length still need verification. Settle the ledger with provider- or gateway-reported usage after the response arrives.

The minimum control has four parts:

1. Normalize omitted candidate counts to 1.
2. Reject counts above the workload’s approved maximum before dispatch.
3. Record requested, effective, and returned candidate counts separately.
4. Reconcile aggregate input, output, and total usage instead of measuring only the selected answer.

This control complements output limits and token quotas. It does not replace them.

Who this is for

This guide is for platform engineers who enforce request policy at an AI gateway, FinOps analysts who reconcile model usage, and product owners whose applications request several answers and choose one.

It is especially relevant when an OpenAI-compatible client can send n, when another provider exposes a field such as candidateCount, or when a gateway translates request parameters across providers. Translation is not proof of identical behavior. The selected route’s current contract remains authoritative.

The control is also useful for evaluation, brainstorming, reranking, and best-of workflows. Those uses may justify more than one candidate, but the exception should name an owner, workload, maximum count, output ceiling, budget, and expiration date.

Key takeaways

• Default the effective candidate count to one at the gateway, not only in application code.
• Require a bounded exception for any workload that needs multiple candidates.
• Reserve output capacity using candidate count and the applicable output ceiling, then settle against reported usage.
• Keep requested, effective, and returned counts distinct so parameter drops and response mismatches are visible.
• Count all generated output reported for the request, not just the candidate shown to the user.
• Verify streaming usage behavior because usage may arrive separately from content chunks.
• Fail closed when a route does not document the candidate parameter or its output-cap semantics.
• Use token quotas as a backstop, while accounting for delayed usage and temporary concurrency overshoot.

Sources checked

• The OpenAI create chat completion reference defines n, states that generated tokens across all choices are charged, and recommends n=1 to minimize cost. It also warns that parameter support can differ by model.
• The Google Gemini GenerateContent reference exposes generation configuration and response usage metadata, including candidateCount and candidatesTokenCount. Those fields support separate measurement of requested fan-out and aggregate candidate output.
• The LiteLLM completion input reference defines n as the number of chat-completion choices and documents request-wide streaming usage. Its provider matrix also shows why operators must verify support for each model and route rather than assuming universal translation.
• The Microsoft llm-token-limit policy reference documents per-key token rates and quotas, actual response-based usage, optional prompt estimation, and concurrency limitations. It provides a concrete gateway pattern for containing aggregate token consumption.

Together, these sources support the control design, but they do not establish that every CometAPI route accepts the same parameter or reports usage in the same shape. Verify the exact route before enabling a multi-candidate workload.

Contract details to verify

Create a short contract record for every approved route. Recheck it when the model alias, provider, gateway version, or client library changes.

A workable policy starts with a global default of one. An exception might allow two candidates for a named evaluation workload, with a fixed per-candidate output ceiling and a short expiration. Avoid unrestricted values controlled by end-user input.

Store only sanitized operational fields. Do not log prompt bodies, generated candidate text, request headers, personal identifiers, or raw tool arguments merely to calculate cost. A sufficient event can look like this:

timestamp=2026-08-13T10:15:00Z
request_id=req-1042
workload_id=support-draft
environment=production
route_id=primary-chat
model_alias=approved-standard
requested_candidate_count=1
effective_candidate_count=1
returned_candidate_count=1
input_tokens=842
output_tokens_total=196
total_tokens=1038
stream_mode=false
usage_source=provider_response
quota_scope=workload
policy_result=allow
policy_reason=within_default

The field output_tokens_total must represent the response’s aggregate output usage when the contract provides it. Do not derive it from the retained candidate’s text alone.

Happy path operator workflow

1. The client omits a candidate parameter or requests one candidate.
2. The gateway identifies the workload and route, normalizes the effective count to one, and confirms that the route contract is current.
3. The gateway reserves estimated input usage plus one output ceiling against the workload budget.
4. The quota check passes, so the request is dispatched once.
5. The response returns one choice and complete usage metadata.
6. The gateway records requested, effective, and returned counts, then replaces the reservation with actual reported usage.
7. The daily control compares aggregate candidate output per request, exception usage, and rejected requests against policy thresholds.

For an approved two-candidate workflow, the same path applies, but the gateway reserves two output ceilings and validates that the response-level usage covers both candidates.

Error-path operator workflow

1. A client requests four candidates while its approved maximum is two.
2. The gateway rejects the request before model dispatch with a stable policy reason such as candidate_count_exceeds_policy.
3. The denial event records the workload, route, requested count, approved maximum, and policy result without recording content.
4. The client receives instructions to request no more than two candidates or obtain a reviewed exception. It must not retry the same disallowed request automatically.
5. If a provider instead returns more candidates than the effective count, or returns content without required aggregate usage, the gateway flags usage_reconciliation_required, preserves the provider request identifier, and does not infer cost from the winning answer.
6. The operator compares gateway evidence with provider usage before releasing the event into the final cost ledger. Replaying the request is not a reconciliation method because it can create more spend.

Failure modes

Only the winning candidate reaches the ledger. An application generates several choices, keeps one, and records only that text’s estimated tokens. The ledger understates output. Use response-level aggregate usage and retain the returned choice count as a separate dimension.

A client library changes the request. A library update can begin sending a non-default count, or a shared helper can expose candidate count directly to callers. Enforce the maximum at the gateway and alert on changes in requested-candidate distribution.

A gateway silently drops the parameter. The application believes it bought several choices, while the route produces one. This is a correctness failure even when it reduces spend. Compare requested and effective parameters, and reject undocumented translation rather than hiding it.

The output ceiling is interpreted incorrectly. One route may apply a generation ceiling differently from another. Multiplying a shared response ceiling as though it were per candidate can over-reserve; assuming a per-candidate ceiling is shared can under-reserve. Record the verified semantics in the route contract.

Streaming usage is missed. Content arrives successfully, but the client disconnects before a final usage event or never requests usage in the supported way. Keep streaming settlement in a pending state until usage is present or independently reconciled. Do not treat a missing final event as zero use.

Concurrency outruns the quota. Microsoft documents that exact consumption may not be known until responses return, so concurrent requests can temporarily exceed a configured token limit. Candidate fan-out increases the size of that exposure. Combine token quotas with concurrency ceilings and conservative reservations.

Retries duplicate fan-out. A timeout or ambiguous network result causes the client to submit the entire multi-candidate request again. Give every attempt its own ledger event, use the route’s supported idempotency behavior where documented, and never erase the first attempt merely because the second succeeded.

Fallback changes parameter behavior. A primary route accepts multiple candidates, while a fallback rejects or ignores the field. Approve candidate behavior for every route in the fallback chain, or disable multi-candidate fallback.

Quota counters have the wrong scope. A team-level limit keyed differently across gateways or regions may not represent a global budget. Microsoft notes that its policy tracks usage independently at each gateway. Record counter scope explicitly and reconcile across scopes before calling a quota organization-wide.

Logs capture unnecessary content. Saving every generated candidate to explain spend increases privacy and retention risk. Cost controls need counts, usage, policy outcomes, and route identifiers, not prompt or completion bodies.

FAQ

Does n=4 make the entire request cost exactly four times as much?

Not necessarily. The OpenAI contract says generated tokens across all choices are charged, so output exposure rises with the number and length of choices. Input charging, cached usage, model pricing, and generation length still depend on the selected route. Use a conservative reservation and settle against actual usage rather than applying a flat multiplier to the final invoice.

Is counting returned choices enough?

No. Choice count reveals fan-out but not output length. Two short candidates may consume less output than one long candidate. The ledger needs aggregate usage plus requested, effective, and returned counts.

When is more than one candidate reasonable?

A bounded evaluation, reranking, or controlled selection workflow may need alternatives. Approval should depend on evidence that the additional candidates improve a defined outcome enough to justify their incremental cost. Give the exception an owner and expiration rather than changing the global default.

What should happen when usage is missing?

Keep the event out of the settled ledger, mark it for reconciliation, and preserve sanitized request and route identifiers. Do not assign zero use, estimate from the selected text, or resend the request solely to obtain metadata.

Does a token quota make a candidate-count limit unnecessary?

No. A quota constrains aggregate consumption, often after some usage is known. Candidate-count policy prevents an unnecessarily expensive request before dispatch and gives product owners a clearer contract.

How is candidate count different from agent-turn count?

Candidate count controls parallel alternatives generated within one model request. Agent-turn controls limit repeated calls across a workflow. Both can multiply spend, but they require different evidence and enforcement points.

Reader next step

Audit one representative week of gateway requests. Identify every field that can request multiple choices, compare requested counts with returned counts, and measure aggregate output usage per request. Set the default and unapproved maximum to one, then create narrow exceptions only where the outcome evidence supports them.

Before approving an exception, count CometAPI tokens before budget approval . After rollout, build token usage evidence for CometAPI budget reviews so reservations, provider-reported usage, and finance records can be reconciled.

When the policy and route checks are ready, Start with CometAPI . Verify candidate-parameter support and usage metadata for each selected route before allowing more than one candidate.