Control Rerank API Costs Before Candidate Lists Expand

Last reviewed: October 30, 2026 (Asia/Shanghai).

Direct answer

Control rerank API costs before dispatch by limiting the number of queries, the candidates submitted for each query, their serialized lengths, and permitted retry attempts. Price that work using the billing meter verified for the exact model and deployment. Keep reranking as a separate cost line from the generation request that follows it.

Cohere’s Rerank API reference defines top_n as the number of returned ranking results. Sending 200 candidates with top_n: 5 still submits 200 candidates for comparison. Reducing the returned list alone does not establish a reduction in billable processing.

The billing contract needs particular care. Cohere’s pricing FAQ defines a search unit as one query with up to 100 documents and says longer documents can contribute multiple billing chunks. The current Rerank API example instead shows meta.billed_units.input_tokens. Those public descriptions do not establish one universal meter for every model and deployment. Confirm which applies before converting candidate counts into a monetary allowance.

Who this is for

This guide is for search engineers, platform teams, and cost owners operating a separately metered reranker between retrieval and answer generation. It focuses on Cohere’s public request and pricing descriptions, checked on October 29 UTC, already October 30 in Asia/Shanghai. Cloud-hosted, dedicated, and gateway deployments require their own applicable pricing evidence.

The admission policy and checks below are recommendations. No authenticated inference, account billing reconciliation, or savings experiment was performed for this article.

Key takeaways

• Limit submitted candidates independently from returned results.
• Count every rewritten query and physical rerank attempt under one business request.
• Version document formatting and length controls because both affect submitted work.
• Verify whether the selected contract bills search units, input tokens, or another unit.
• Retain unresolved usage after a timeout and admit any replacement separately.
• Approve a smaller candidate set only when retrieval quality remains acceptable.

Sources checked

Three primary public pages returned HTTP 200 during this review:

• Cohere: Rerank API (v2) defines documents, top_n, document truncation, optional response metadata, and an example containing billed input tokens.
• Cohere: Pricing , specifically its Rerank search-definition FAQ, defines one query with up to 100 documents as a search unit and describes a 500-token billing-chunk threshold that includes query length. This is a pricing description to qualify against the selected contract.
• Cohere: Best Practices for using Rerank distinguishes model-specific document handling, documents v5’s default max_tokens_per_doc of 4,096, and recommends representative queries when selecting a relevance threshold.

These are maintained technical and pricing references for an evergreen budgeting task. Their access date does not establish a new product announcement. Recheck the applicable meter, model behavior, and rate before a launch or material configuration change.

Contract details to verify

Separate the three candidate counts

Record how many candidates retrieval returns, how many the application sends to reranking, and how many ranked results it retains. Give those counters explicit names instead of calling all three top_k.

For a concrete failure, suppose retrieval previously supplied 40 passages and an agentic search change now combines five searches into 200 passages. The interface still displays five results, and top_n remains five. That visible stability hides a fivefold increase in submitted candidate count. It does not prove a fivefold charge: billing groups, document lengths, and the selected meter determine the financial effect.

Deduplicate repeated candidate identities before dispatch when they represent the same permitted evidence. Preserve distinct passages that contain different qualifications. Apply access filters before any external processing. A cost optimization must preserve the retrieval system’s data-access rules.

Use the separate RAG context-budget workflow after ranking. A generation context ceiling cannot retroactively constrain reranking work already submitted.

Qualify the meter for one model and deployment

Create a dated record containing the endpoint, exact model ID, deployment type, currency, billing unit, rate denominator, applicable price evidence, and observed usage fields. Read the actual production contract as well as public documentation.

Under the pricing FAQ’s search-unit description, document count can expand when a document and query exceed the stated 500-token boundary. The FAQ says each resulting chunk counts as an individual document for that search. Physical input-array length alone therefore may understate the billing quantity. Query length belongs in this check too.

Keep this billing definition separate from model context and inference chunking. Cohere’s best-practices page describes different processing behavior across model families. A model’s context window is not a definition of a billable search unit. Likewise, a document truncation parameter does not independently establish how the account will meter the retained input.

The current API example’s billed-input-token field is a further reason to inspect real usage for the selected model. Preserve the returned meter names and values. If the response and approved pricing evidence disagree, hold financial approval while the owner resolves the mapping. Do not replace a missing search-unit value with zero or assume a generic token count is the billed quantity.

Once the mapping is verified, calculate in that unit:

estimated_attempt_cost = estimated_billed_quantity * approved_rate_per_unit
request_exposure = settled_rerank_cost + unresolved_rerank_reservations
admit only if request_exposure + attempt_reservation <= rerank_allowance

Normalize a rate quoted per thousand search units or per million tokens before applying the first line. Keep an estimate distinct from calculated usage cost and final billing reconciliation. A hard spending boundary requires a defensible maximum for the covered charge; an empirical average supports a soft planning allowance.

Bound query expansion and candidate length

Give each user action one rerank allowance covering all query rewrites, fallback searches, and attempts. A limit of 80 documents per call permits 320 submitted documents across four calls. Setting top_n: 5 on each does not reduce that submitted total.

For illustration, if a verified search-unit contract assigns one unit to each of those four calls, the planned quantity is four units. If the approved rate is R per unit, the planned rerank cost is 4 * R. This conditional arithmetic uses no provider price and makes no claim about the meter applied to a particular v5 account.

Choose explicit candidate and length ceilings below the endpoint’s capacity. The API reference recommends no more than 1,000 documents for optimal performance; that recommendation is not a workload budget. For v5, the best-practices page documents automatic truncation at max_tokens_per_doc, defaulting to 4,096. Set and record the approved value rather than allowing an SDK or model change to choose it silently.

Validate the final serialized strings, including titles and metadata. Keep important fields within the accepted input and test documents whose relevant evidence appears near the end. Sending a whole document that is later truncated can remove the passage that retrieval intended to rank. If the application prepares passages itself, version their extraction and ordering rules alongside the length policy.

Happy-path operator workflow

1. Freeze one workload’s model, meter mapping, rate revision, query limit, candidate ceiling, length policy, retry allowance, and quality floor.
2. Collect the permitted candidates, remove exact duplicates, and build the actual serialized rerank request. Count its queries, documents, and relevant length measures.
3. Estimate the billable quantity under the verified contract and reserve the attempt’s approved exposure. Use the existing atomic budget reservation ledger when workers share an allowance.
4. Dispatch once under a distinct attempt ID. Record returned indexes and validate their mapping to the submitted candidate list before assembling evidence for generation.
5. Capture the actual usage fields without renaming their units. Calculate cost using the matching rate revision and reconcile later with billing evidence.
6. Track total reranking and generation cost per accepted answer. Preserve all attempt costs when a search or answer fails its quality check.

Keep request ID, attempt ID, model, deployment, policy revision, query count, submitted-document count, length summary, top_n, returned count, meter name, billed quantity, cost state, and quality outcome. Store raw queries, document text, credentials, and sensitive document labels outside routine cost telemetry.

Error-path operator workflow

If the prepared request exceeds its candidate, query, or financial allowance, stop before dispatch. Use only a preapproved pruning or fallback policy. Recheck the evidence floor after pruning; an inexpensive answer with inadequate evidence is a failed outcome.

If a response times out, retain that attempt and its unresolved reservation. A permitted retry gets another identity and another admission decision under the same user-action allowance. Ensure SDK retries are either controlled by this dispatcher or included in its approved exposure. A retry that resets the business-request counters defeats the limit.

If usage is missing or uses an unexpected meter, mark cost pending and retain the response reference for reconciliation. Investigate the model, deployment, response schema, and pricing record before widening traffic. The public examples do not justify inventing a conversion between search units and billed input tokens.

Verification before rollout

First test local policy enforcement with a counted fake dispatcher. Then run a small approved account canary to qualify the actual meter. These are proposed acceptance checks, not reported test results.

For the canary, compare at least two candidate ceilings on the same representative queries, corpus revision, and relevance judgments. Keep generation settings fixed when measuring end-to-end cost. Cohere’s best-practices guide recommends 30–50 representative queries for selecting score thresholds; use an appropriate domain evaluation rather than treating one universal score as a quality guarantee.

Measure whether the required evidence reaches the retained results, answer acceptance, latency, and reconciled cost. A lower candidate ceiling saves useful money only when the resulting service still clears its quality floor. Add the total to the workload unit-cost scorecard .

Failure modes

The returned list becomes the spend counter. Five retained results conceal a large submitted list. Enforce the candidate ceiling against the outgoing documents array.

A long query expands the billable work. The team counts document text alone under a search-unit contract. Include query length in the applicable billing calculation and qualify boundary cases.

A model upgrade keeps the previous meter. The client changes model IDs while its calculator still expects search units. Require a new meter and rate approval, including actual usage evidence.

Quality loss triggers automatic widening. A weak result causes repeated larger searches. Bound expansion under one allowance and return the approved insufficient-evidence outcome when it is exhausted.

Only generation savings appear in the report. Smaller context looks economical while reranking and retries cost more. Compare the full request cost using the same accepted-answer denominator.

FAQ

Does lowering top_n reduce the rerank charge?

It reduces the requested number of returned results. The cited API definition does not establish a billing reduction from that change alone. Control submitted work and verify the actual meter.

Are 100 submitted documents always one search unit?

The pricing FAQ defines up to 100 documents per query, but says longer documents can expand into multiple billing chunks. Confirm that this search-unit definition applies to your model and deployment before using it.

Is max_tokens_per_doc a spending cap?

It bounds document content processed under the documented v5 behavior. A monetary ceiling also needs candidate, query, attempt, and billing-unit controls. Its default does not establish an approved budget.

Can reranking lower total RAG cost?

It can help select less context for generation. Compare that benefit with rerank charges, retries, and answer quality on the same workload before approving an economic claim.

Reader next step

Choose one reranked workload and record its submitted-candidate count beside top_n. Verify the model’s actual billed meter, approve one query-and-candidate allowance, and test local denial before expanding traffic. Then compare two candidate ceilings on a fixed quality evaluation and reconcile the canary’s total cost. Keep the smaller policy only when the evidence and accepted answers remain adequate.