Last reviewed: 2026-08-05

Direct answer

RAG context token budgets should use two controls: a limit on retrieval candidates and a hard ceiling on the retrieved tokens admitted to the model request. A result-count limit constrains search fan-out, but it cannot guarantee cost because chunks can differ substantially in length. The token ceiling controls the final payload that becomes model input.

At request time, reserve room for the fixed instructions, the user input, tool metadata, and the expected output. Retrieve only the configured number of candidates, apply filters and reranking, then admit chunks in relevance order until the retrieval-token ceiling is reached. If the remaining evidence is too weak to answer, use a defined error path instead of silently increasing the ceiling.

This approach matches the available platform controls. The OpenAI file search documentation says limiting retrieved results can reduce token usage and latency, while warning that quality can fall. The Microsoft guidance on RAG in Azure AI Search identifies token constraints as a core RAG challenge and recommends concise, highly relevant results rather than exhaustive document dumps.

A practical admission rule is:

retrieval_token_cap =
  min(
    workload_retrieval_cap,
    model_context_limit - fixed_input_tokens - output_reserve
  )

The workload cap should come from observed unit cost and quality, not from the largest context window available. A larger window is capacity, not permission to fill every request with more retrieved text.

Who this is for

This guide is for platform engineers who operate retrieval-augmented generation, application teams that own answer quality, and FinOps or budget owners who need predictable cost per request. It is most useful when a search layer can return a variable number of differently sized chunks, when reranking narrows a larger candidate set, or when the same RAG service supports workloads with different cost and evidence requirements.

The method applies whether retrieval is hosted by a model provider, a cloud knowledge-base service, or an application-managed search system. Provider field names differ, so the operating contract must record which limit applies at each stage.

Key takeaways

  • Set both a candidate-result limit and a final retrieved-token limit. Neither one substitutes for the other.
  • Reserve input and output capacity before admitting retrieved chunks.
  • Define a minimum evidence floor and an explicit over-budget path; do not let the application raise limits automatically.
  • Measure candidate count, admitted chunks, retrieved tokens, total input tokens, quality outcome, latency, and estimated request cost together.
  • Use the same request representation for preflight counting that the model will receive. The preflight token-count controls provide a related review pattern.
  • Evaluate the guardrail against workload-level economics with a unit-cost scorecard , not just an aggregate monthly token total.

Sources checked

Together, these sources support the operating pattern, but they do not establish one universal field name, default, price, tokenizer, or quality threshold. Those details remain workload and provider contracts to verify.

Contract details to verify

Map each stage before setting a cap

Document the stages of the live request path: query preparation, candidate retrieval, metadata filtering, reranking, context serialization, model input, and any retry or fallback. A single setting called top_k can hide several different meanings. It might control candidates from one vector query, final results after fusion, or chunks passed to generation. Record the stage and observed behavior instead of inferring it from the name.

For OpenAI file search, verify the active result-limit field and inspect the exact content eventually included in the model request. For Amazon Bedrock, numberOfResults is specifically the number of source chunks to retrieve; the documented 1-to-100 range should not be copied into another provider contract. Bedrock also exposes filtering and reranking configuration, so record whether the ledger counts retrieval, reranking, and generation as distinct operations. Microsoft describes top-k or top-n limits and minimum thresholds as classic RAG controls; verify which control your implementation actually uses.

At minimum, the workload contract should contain:

  • candidate_limit: the maximum results requested from the retrieval stage.
  • reranked_limit: the maximum candidates retained after reranking.
  • retrieval_token_cap: the maximum serialized retrieved context admitted to generation.
  • fixed_input_allowance: the expected space for instructions, user input, and tool metadata.
  • output_reserve: capacity held for the response rather than consumed by retrieval.
  • minimum_relevance: the workload’s evidence threshold, using the score semantics of its retrieval system.
  • minimum_evidence: the least evidence required before generation is allowed.
  • fallback_mode: the permitted action when retrieval is empty, weak, unavailable, or over budget.
  • owner and review_trigger: who approves changes and what variance opens a review.

Defaults should be workload-specific. A short policy lookup and a research assistant do not have the same evidence needs, even if they share an index and model.

Happy-path operator workflow

Use this sequence for an ordinary request:

  1. Count or estimate the fixed input before retrieval and subtract the output reserve from the request allowance.
  2. Send the retrieval query with the approved candidate limit and required metadata filters.
  3. Record the number of candidates returned. Apply the approved relevance threshold and reranker.
  4. Serialize the surviving chunks exactly as they would appear in the model input, including separators and source labels.
  5. Count that serialized context. Admit chunks in rank order, stopping before the retrieval-token ceiling.
  6. Confirm that the admitted set meets the minimum-evidence rule. If it does, make one generation request.
  7. Capture the model’s reported usage when available and compare it with the preflight count. Persistent variance is a metering defect to investigate.
  8. Record the quality result from a sampled evaluation or user-visible outcome, then review cost and quality together.

Consider an illustrative workload with a candidate limit of 20, a reranked limit of 6, and a retrieval cap of 3,000 tokens. If five admitted chunks serialize to 2,640 tokens and meet the evidence floor, the request follows the allow path. These values are an example, not provider defaults.

Use sanitized, structured records such as:

event_time: 2026-08-05T00:00:00Z
request_id: rag-042
workload_id: support-rag
retrieval_mode: hybrid
candidate_limit: 20
candidates_returned: 18
reranked_limit: 6
chunks_admitted: 5
retrieval_token_cap: 3000
retrieved_context_tokens: 2640
total_input_tokens: 3840
output_tokens: 310
estimated_request_cost_usd: 0.021
budget_action: allow
quality_check: pass
error_code: none
query_text: '[REDACTED]'
source_content: '[REDACTED]'

Keep raw queries, retrieved passages, personal data, and confidential document labels out of the cost log. Join to separately governed diagnostic data only when an authorized investigation requires it. The cost record needs counts, policy decisions, and quality signals, not the source text itself.

Error-path operator workflow

Suppose the reranker returns six chunks that serialize to 4,350 tokens against a 3,000-token ceiling. Remove the lowest-ranked chunk, recount the exact serialized input, and repeat until the request fits. Do not truncate a chunk blindly if doing so could separate a claim from its qualifier or citation.

If the reduced set falls below the minimum-evidence rule, stop before generation and return the workload’s controlled insufficient-evidence response or route the request to review. Record context_over_budget or evidence_floor_not_met with the measured counts. Do not raise the cap automatically, because that converts an exception into unreviewed budget drift.

If retrieval times out, use only the fallback approved for that workload. A bounded retry can be appropriate, but it must be counted as another operation and governed by the same request budget. The retry evidence guide provides a related accounting pattern. If token usage or retrieval counts are unavailable, mark the request metering_unavailable; do not classify it as within budget merely because the fields are missing.

Failure modes

Treating result count as a token ceiling. Five large chunks can exceed the input of twenty small chunks. Enforce the cap after context serialization, not only at retrieval time.

Counting before formatting. Source labels, separators, metadata, and repeated instructions can enlarge the model input. Count the representation that will actually be sent. Compare preflight and reported usage so formatting changes do not remain invisible.

Reranker fan-out missing from the ledger. A system may retrieve many candidates and pass only a few to generation. The final model context can be within budget while a separately metered reranking stage is not. Verify the provider contract and record candidate and admitted counts separately.

Chunk-size drift after reindexing. A parser, overlap, or segmentation change can make the same result limit produce more context. Version the indexing configuration and reopen the budget review when chunk-size distributions change.

Duplicate or overlapping evidence. Neighboring chunks can repeat the same text, consuming tokens without adding support. Deduplicate or penalize heavy overlap before admission, then verify that citations still point to usable evidence.

Thresholds that are too permissive. Weakly related chunks fill the budget and can crowd out stronger evidence. Track relevance outcomes and unsupported-answer rates alongside cost.

Thresholds that are too strict. Aggressive pruning can create empty retrievals or omit necessary qualifiers. The error path must surface insufficient evidence instead of encouraging the model to answer from unrelated prior knowledge.

Automatic cap escalation. Raising the limit whenever evidence is weak removes the guardrail precisely when retrieval quality is failing. Require a reviewed policy change or route the request to a defined fallback.

Retries counted only once. Retrieval, reranking, and generation attempts can repeat after timeouts or transient failures. Give each attempt an identifier, record its measured usage, and aggregate all attempts into the request’s unit cost.

Average-only monitoring. A stable mean can hide a costly tail caused by a small class of broad queries. Review distributions by workload and outcome, including over-budget blocks, empty retrievals, and high-context requests.

Sensitive content in cost telemetry. Logging full queries and chunks makes a billing record unnecessarily sensitive. Store sanitized counts and decisions in the cost ledger, with protected diagnostic evidence handled separately.

FAQ

Is a lower result limit always cheaper?

It lowers the maximum number of results available for downstream context, and OpenAI documents reduced token usage as a potential benefit. It does not guarantee a particular cost because chunk lengths, formatting, reranking, retries, and the final admitted set can vary. Measure the serialized context and the provider-reported usage.

How should a team choose its first retrieval-token cap?

Start with representative successful requests. Measure fixed input, retrieved tokens, total input, quality outcome, and cost per useful result. Choose the smallest cap that maintains the workload’s approved quality floor, then test error and edge cases. Do not derive the cap only from the model’s maximum context capacity.

Can reranking replace a token cap?

No. Reranking improves ordering and can help select a smaller evidence set, but it does not guarantee that the selected chunks fit a token ceiling. Apply the cap after reranking and after final serialization.

Should every workload use the same minimum evidence rule?

No. A narrow lookup may need one authoritative passage, while a comparative or policy-sensitive answer may require multiple sources or qualifiers. Define the evidence rule with the workload owner and test it against known good and known insufficient cases.

Can character count stand in for token count?

Character count can be a conservative early filter, but it should not be the final control when a compatible token counter is available. Tokenization varies with content and model conventions. The admission decision should use the closest available representation of the actual request.

Does a larger context window solve this cost problem?

It reduces the chance of exceeding capacity, but it does not establish an economical request size. Microsoft explicitly warns that sending exhaustive document collections wastes tokens and can degrade quality. Keep a workload cap even when the model can accept more context.

Reader next step

Choose one high-volume RAG workload and write down its candidate limit, reranked limit, retrieval-token cap, output reserve, evidence floor, and fallback mode. Add the sanitized fields above to its cost record, then run a representative set through both the happy and error paths. Compare unit cost with quality before changing production defaults.

Use the AI API budget change-control workflow to record the approved baseline, owner, evidence, and review trigger. The immediate goal is simple: no RAG request should exceed its context budget without producing a visible, reviewable decision.