Cap CometAPI Shadow Traffic Before Mirroring Production

Last reviewed: November 1, 2026.

Direct answer

A CometAPI shadow traffic budget needs a separate allowance for every experimental model call, retry, and evaluator call. Use the mirror percentage to select eligible traffic, then require a shared budget reservation before each billable attempt. Stop new shadow dispatch when the allowance is committed, the experiment expires, or an operator disables it. Production traffic continues under its existing policy.

Traffic mirroring creates additional work even when the experimental answer never reaches a user. Istio’s mirroring guide describes copying live requests outside the primary response path and discarding mirrored responses. Envoy’s route-mirroring example likewise sends a request to both the primary and mirror clusters while returning only the primary response. Discarding an answer does not establish that the upstream computation was free.

The design below puts an application-controlled shadow adapter in front of CometAPI. It measures incremental experiment cost separately from the production baseline. Sampling, financial admission, and experiment shutdown each have a specific job; all three need to work before live copying begins.

Who this is for

This guide is for platform engineers and FinOps owners comparing a candidate model against live production requests. It assumes one approved text-only request class, a separate shadow executor, and control over every outgoing experimental attempt.

These are recommended implementation rules and acceptance checks. No authenticated CometAPI billing experiment, mesh deployment, or production traffic test was performed for this article. The numerical example uses hypothetical prices. Select actual model rates and verify endpoint behavior before approving a run.

Key takeaways

• A fixed mirror percentage produces more calls when production volume grows.
• Include every candidate, retry, and model judge in incremental spend.
• Reserve funds before dispatch and retain commitments when usage is unresolved.
• Give the experiment an expiry and enforce its state again when queued work starts.
• Keep shadow execution outside user responses, production writes, and tool execution.
• Verify the stop control with queued and outstanding work, then reconcile actual usage.

Contract details to verify

Calculate the additional work

For one request class, let N be production requests during the experiment, e the eligible fraction, and p the mirror fraction among eligible requests. If the sampler selects each eligible request with probability p, the expected selected count is:

expected_selected_requests = N × e × p
incremental_model_cost = sum(attempt_count_c × average_attempt_cost_c)
incremental_total_cost = incremental_model_cost
  + evaluator_cost + added_storage_and_processing_cost

The sum covers each candidate and retry class separately. It excludes production calls that would occur without the experiment. Include additional baseline generations if the experiment creates them. Reusing a permitted production response for comparison does not itself create another baseline generation.

CometAPI’s request-cost guide says to combine the selected model’s price with the endpoint’s billing units and compare estimates with actual usage afterward. For token-priced text calls, calculate input and output components separately. Its example uses max_completion_tokens to limit output; verify the selected model supports the intended limit. Character-count heuristics are planning estimates, not dependable token ceilings.

Consider this hypothetical daily forecast:

The expected sample contains 8,000 requests. Each candidate attempt averages (1,500 × 2 + 300 × 6) / 1,000,000 = $0.0048. Two candidates therefore cost an estimated $76.80. Another 400 retry attempts at that same assumed average add $1.92; 2,000 judge calls averaging a hypothetical $0.001 add $2.00. The model subtotal becomes $80.72 before storage, comparison processing, or other overhead.

Those are forecast averages, not reservation amounts or CometAPI prices. Random selection need not yield exactly 8,000 requests, and longer inputs change cost. If otherwise identical eligible traffic doubles, expected candidate-generation spend doubles too. A 2% setting cannot enforce an $85 allowance.

Put financial admission before CometAPI dispatch

Define one experiment record containing its owner, approved currency amount, start and expiry times, permitted workload class, candidate model IDs, price revision, input and output limits, attempt caps, and stop-control generation. Give each candidate attempt an identity derived from the experiment, opaque source-event ID, candidate, and attempt number. A duplicate queue delivery must resolve to the existing attempt record.

Follow the site’s atomic budget reservation workflow for the shared financial transaction:

available = approved_allowance - settled_cost - unresolved_reservations
admit only when this_attempt_reservation <= available

All workers must use the same authority or disjoint allowances whose sum fits the experiment. A per-process counter is insufficient for a shared limit. Reserve using a conservative, verified bound for the exact request, including supported input limits, output limits, and separately charged operations. If the application cannot bound a request class, exclude it from this capped experiment or explicitly approve a soft forecast policy.

Commit the reservation before issuing one dispatch claim. Charge each permitted retry and judge attempt through admission too. Disable hidden SDK retries, or make their maximum exposure part of an explicitly accounted dispatch contract. An average request cost cannot replace the reservation bound.

When reliable usage arrives, settle the calculated amount and release the corresponding reservation in one idempotent transaction. Keep a timed-out or disconnected attempt unresolved until evidence supports settlement. A local cancellation or expired worker lease does not establish zero upstream usage. Retain the commitment, or convert it to a documented conservative accrual when the approved accounting policy requires that treatment.

Keep the sampler separate from the executor

Use this application flow:

approved request class → sampling decision → bounded shadow queue
  → expiry and experiment-state check → shared spend reservation
  → one dispatch claim → CometAPI call → usage settlement

The primary response must not wait for shadow completion. When the queue is full or financial admission is unavailable, skip experimental work according to the declared policy and record why. Avoid an unlimited queue: delayed copies can accumulate after the intended test window and create a later spending burst.

Recheck expiry and the current stop generation at the controlled dispatch gate, including for retries and judges. Serialize stop-state changes with dispatch permission in that authority. Work that has not crossed the gate loses permission to start. Treat attempts that already crossed it as potentially sent, even if their network writes are delayed, and retain their commitments. A flag update cannot recall an external request. Test this race explicitly; reading an enabled flag only when enqueueing does not stop old queued jobs.

With Istio, configure the intended mirrorPercentage.value explicitly. Its guide says omission mirrors all traffic in that configuration. It also documents appending -shadow to the mirrored Host/Authority header. Inspect the actual request your adapter receives and construct the outbound CometAPI request from approved server configuration rather than forwarding that mirror host blindly.

Envoy’s example supports selecting a mirror cluster through a request header and warns that this belongs in a trusted environment. Keep destination selection under deployment control; untrusted callers must not choose a candidate route or activate paid shadow work through an inbound header.

An application adapter can perform the required budget check even when the mesh supplies copies. Confirm that no other mirror layer also calls the model. If both an edge proxy and an application worker independently duplicate requests, one sampled user action can produce more attempts than the forecast assumes.

Define what evidence the mirror produces

Istio’s documented fire-and-forget mirror discards responses. That can exercise transport and capacity, but it does not by itself create a paired model-quality dataset. If comparison requires responses and usage, make the shadow adapter own the CometAPI call and capture the permitted results before returning to the mirror sender.

Use opaque comparison IDs and content-free cost events: experiment, candidate, attempt, sampling-policy version, admission result, price revision, reservation, reported usage, calculated amount, and settlement state. Keep prompts, completions, authorization headers, and credentials out of routine cost logs. Store any approved comparison content separately with access and deletion controls.

Start with text-only requests that cannot execute tools or change production state. A successful shadow response must not populate a production answer cache, send a notification, or become an automatic fallback answer. Additional evaluators need their own allowance and concurrency limits. Use the evaluation-budget guide to plan the later quality decision; this traffic budget does not establish a statistically sufficient sample.

Smoke-test workflow

First use a counted fake model and a disposable ledger. These are acceptance criteria to run, not reported test results.

For the happy path, approve one small allowance, one candidate, and one text-only class. Verify the endpoint-specific output limit, send a bounded synthetic sample, and join every admitted attempt to either settlement or an unresolved commitment. Inspect the cost of invalid responses as well as valid ones. Then enable a small live sample only after the isolation and stop checks pass.

For the error path, disable new shadow admissions, invalidate queued dispatch permission, and stop new retries and judges. Reconcile outstanding calls and retain unresolved value before closing the experiment. Restart under a new approved experiment version if the route, rate, covered request class, or allowance changes. Do not drain an expired backlog merely because it was selected earlier.

Failure modes

A percentage is treated as a spending cap. A production burst increases mirrored volume. Retain the sampler for selection and use financial admission to enforce the experiment’s verified scope.

Only successful comparisons enter the ledger. Timeouts, invalid output, and failed judges disappear from the report. Record every dispatched attempt and distinguish cost per dispatched attempt from cost per usable comparison.

Retries are mirrored again. An internal retry reenters the original sampling path and creates new candidate jobs. Mark experiment-origin traffic internally, prevent recursive copying, and enforce unique attempt identities.

The stop switch leaves paid work queued. Admission was checked only at enqueue time. Enforce expiry and stop state at the dispatch boundary and account for calls already started.

Candidate work uses shared production capacity without limits. Separate concurrency and connection resources where practical, and skip shadow work when those limits are reached. Independent response routing alone does not remove competition for CPU, connections, or provider capacity.

A smaller sample is reported as a successful evaluation. Budget rejection or queue drops change the collected cohort. Report selected, admitted, skipped, dispatched, usable, and unresolved counts by request class. Insufficient evidence should remain insufficient when the allowance ends.

Sources checked

Three primary public references were fetched successfully on November 1, 2026:

• CometAPI: Estimate request cost before calling a model , displaying a June 23, 2026 modification date: model-specific billing units, input/output estimates, an output-limit example, and reconciliation with actual usage.
• Istio: Mirroring , served by the current documentation site identifying Istio 1.31.3: out-of-band copying, discarded responses, explicit percentage selection, omission behavior, and the mirrored Host/Authority suffix.
• Envoy: Route mirroring policies , served as development documentation: primary and mirror routing, primary-only client responses, and the warning about header-selected mirror destinations.

These maintained technical references support an evergreen operating task. The reservation design, shutdown procedure, arithmetic, and test matrix are application recommendations derived from the documented behavior. Verify mesh behavior against your deployed release; no development-version configuration is prescribed here.

FAQ

Does discarding the mirrored answer eliminate its model cost?

Response disposal is a routing behavior. Budget every experimental attempt that reaches CometAPI and use actual usage and billing evidence to determine its cost. A missing client-visible answer supplies no proof of zero consumption.

Is a dedicated API key enough to enforce this allowance?

A separate key helps attribution. Its enforcement semantics still need verification. Route all covered shadow attempts through the same experiment admission authority, including retries, judges, and queued work.

Can the sample continue until the results look convincing?

The experiment ends at its allowance, expiry, or operational stop. If evidence is insufficient, close it with that result and approve another plan separately. Define the quality-analysis procedure before inspecting results.

What changes when we compare three candidates?

Each selected request can create three candidate attempts before retries and judges. Price each candidate separately and give every attempt its own reservation. The production mirror percentage alone does not express this multiplication.

Reader next step

Write one experiment record with an owner, allowance, expiry, candidate, permitted request class, output limit, and attempt cap. Prove the two-worker admission race and the queued-work shutdown check using a fake model. Then Start with CometAPI for a bounded synthetic run, reconcile every attempt, and review that evidence before copying live traffic.