Last reviewed: 2026-08-12

Direct answer

AI conversation history cost controls should operate on rendered input tokens, not message count alone. Before each turn, measure the complete input the selected model will receive, compare that estimate with the workload’s usable token budget, and compact only when a documented threshold is crossed. After compaction, count again and verify that the user goal, accepted constraints, unresolved actions, and required safety instructions survived.

This matters because multi-turn state can become recurring input. The Google token-counting documentation says API cost is determined in part by input and output token counts and shows that usage for a continued interaction can include tokens from earlier turns. The Amazon Bedrock Converse documentation likewise instructs clients to include conversation messages in subsequent requests when maintaining context. A conversation that gains context every turn can therefore consume progressively more input even when each new user message is short.

Compaction provides a controlled alternative to replaying everything. The OpenAI compaction guide describes a configurable compact threshold and a compacted state item that carries necessary prior context into later turns using fewer tokens. It also warns that state-handling rules differ between stateless input-array chaining and chaining with a previous response identifier. Treat those mechanics as part of the provider contract, not as interchangeable implementation details.

A practical threshold model

Start with a team-defined usable input budget after reserving room for output, tools, and a safety margin. For illustration, a workload with a 20,000-token usable input budget could warn at 12,000 tokens, compact at 14,000, and stop automated execution at 17,000. These are policy examples, not provider limits. Set actual thresholds from the selected model’s documented limits, observed output size, latency objective, and acceptable per-turn cost.

For every turn, record four measurements:

  1. Full preflight input estimate for the exact model and request shape.
  2. Estimated fixed input from system instructions and tool definitions.
  3. Estimated new input from the current message, new attachments, and new tool results.
  4. Actual usage returned after the request.

A useful diagnostic estimate is:

  • Estimated history tokens = full input estimate minus fixed input minus new input.
  • History share = estimated history tokens divided by full input estimate.
  • Token reduction = pre-compaction estimate minus post-compaction estimate.

Do not present the history estimate as a provider billing field. It is an internal allocation derived from measured totals. Reconcile it with actual response usage, because preflight counters and billable usage may differ.

Happy-path operator workflow

  1. Define the usable input budget, warning threshold, compaction threshold, hard stop, and required quality anchors for one workload and model version.
  2. Count the complete request before sending it. Use the same model, system instructions, tools, messages, and attachments planned for generation.
  3. If the count is below the warning threshold, send the request and reconcile estimated versus actual usage.
  4. If it is between warning and compaction thresholds, send it but flag the conversation for closer observation.
  5. If it crosses the compaction threshold, invoke the approved compaction path once. Preserve provider-returned state exactly where the provider contract requires it.
  6. Count the resulting request again. Continue only if it falls below the hard stop and leaves the required output headroom.
  7. Run quality-anchor checks. Confirm the compacted state retains the user’s objective, decisions already made, current constraints, pending actions, and safety requirements.
  8. Send the next turn, capture actual usage and latency, and compare the result with the pre-compaction baseline.

The Anthropic token-counting documentation is a useful example of why request shape matters: its counter accepts structured messages, system prompts, tools, images, and PDFs. It also characterizes the result as an estimate and recommends counting against the model that will actually handle the request.

Who this is for

This control is for platform engineers who maintain multi-turn assistants, FinOps practitioners who reconcile token spend, and budget owners who need a defensible reason for when context is retained, compacted, or stopped. It is especially relevant to support assistants, research workflows, coding agents, and tool-using systems whose conversations can continue long after the initial prompt.

The control needs both technical and business ownership. Engineering owns state integrity and failure handling. The cost owner defines the per-turn envelope and escalation rule. The workload owner approves the quality anchors that compaction must preserve.

Key takeaways

  • Count the rendered request for the exact model before each turn; do not infer cost from turn count or character count alone.
  • Separate fixed prompt, new-turn, attachment, tool, and estimated history contributions so operators can explain growth.
  • Reserve output and tool headroom before setting compaction thresholds.
  • Compact once at a controlled threshold, recount, and validate critical state before continuing.
  • Reconcile preflight estimates with actual provider usage rather than treating estimates as invoices.
  • Stop automatic execution when counting, compaction, or quality validation fails.
  • Log measurements and decisions, not raw conversation content.
  • Recalculate thresholds after model, tokenizer, tool-schema, or state-management changes.

Sources checked

  • OpenAI Compaction was checked for automatic and standalone compaction behavior, threshold triggering, compacted-state handling, and the distinction between stateless chaining and previous-response chaining.
  • Anthropic Token counting was checked for pre-request counting, supported structured inputs, estimate caveats, and model-specific counting guidance.
  • Google Understand and count tokens was checked for input counting, response usage categories, multi-turn accounting, and context-window considerations.
  • Amazon Bedrock Inference using Converse API was checked for message-based conversation state, subsequent-request history handling, request metadata, and cache-checkpoint availability.

Together, these sources support the operating model: measure the real request shape, understand how state is carried, compact through an approved mechanism, and verify actual usage afterward. They do not establish one universal threshold or billing formula. Those values must come from the workload policy and current provider contract.

Contract details to verify

Before enabling compaction in production, record the following details for every routed model and provider:

  • The input limit, output limit, and whether the published context window combines them.
  • Which parts of the request are tokenized, including system instructions, tool definitions, tool results, images, documents, and provider-managed state.
  • Whether the preflight count is exact or estimated and which response fields represent actual input, output, cached, thinking, or tool-use tokens.
  • Whether conversation state is replayed by the client, referenced through a provider identifier, or represented by a compacted state item.
  • The supported compaction trigger, minimum or maximum threshold, and behavior when compaction cannot reduce the request enough.
  • Whether earlier input items may be removed after compaction. OpenAI’s documented rule differs depending on whether the client uses stateless array chaining or a previous response identifier.
  • How compaction events, cached tokens, and provider-managed state are billed under the current agreement.
  • The retention and data-handling terms for original messages, compacted state, logs, and invocation metadata.
  • Rate limits for counting and generation, including whether a counter outage should block, degrade, or reroute the workload.
  • The approved fallback model and whether its tokenizer or context rules require a fresh count.

Use contracted rates to turn measured token classes into cost estimates. A simple economic test is to divide the measured cost of a compaction event by the expected savings on each later turn. The result is the number of successful reuse turns needed to break even. Do this with monetary cost after applying the applicable rate to each token class; do not compare unlike token categories as if their prices were identical.

Sanitized logging fields

A cost-control event can be useful without retaining prompts or outputs. A sanitized record might look like this:

event_time: 2026-08-12T00:00:00Z
workload: support_assistant
conversation_ref: conv_4821
turn_number: 18
model: configured-model
policy_version: history-v3
input_tokens_estimated: 14320
input_tokens_actual: 13980
output_tokens_actual: 620
cached_tokens_actual: 0
tool_tokens_actual: 110
compaction_threshold_tokens: 14000
compaction_triggered: true
post_compaction_tokens_estimated: 8420
quality_anchor_status: pass
result: completed
error_class: none
retry_count: 0
latency_ms: 2410

Keep raw prompts, outputs, attachments, document text, tool arguments, user identifiers, and opaque provider state out of the cost ledger. Use a short internal conversation reference that cannot be used to retrieve content without access to the separate application system. Apply the same retention and access controls used for other operational telemetry.

Failure modes

History grows unnoticed. A dashboard tracks new user tokens but not the full rendered request. Per-turn input rises while the visible prompt remains short. Fix this by counting the complete request and graphing input tokens by turn number and conversation age.

The estimate is treated as the invoice. A preflight counter can differ from actual usage. Anthropic explicitly describes its count as an estimate. Reconcile every sampled request with response usage and alert on sustained variance rather than forcing equality on every call.

State is carried twice. A client combines provider-managed conversation state with a manually replayed history. Even when the request succeeds, token attribution and state ownership become unclear. Choose one documented state path per integration and test the rendered request after SDK changes.

Compaction drops a critical constraint. The next response forgets an accepted decision, safety restriction, or pending action. Token reduction alone is not a pass condition. Require quality-anchor checks and keep a non-sensitive inventory of which state categories must survive.

Turn count becomes a proxy for size. Ten tool-heavy turns can be larger than fifty short text turns. Trigger on measured tokens and usable headroom, with turn count used only as a secondary diagnostic.

Attachments and tools are excluded. Text history appears within budget while tool schemas, tool results, images, or documents consume the remaining context. Count the full structured request. Both Anthropic and Google document token accounting beyond plain text.

Compaction loops. A failed post-compaction count triggers another compaction attempt, which triggers another retry. Allow one automated compaction attempt per turn. After that, stop or use an approved degraded path.

A model change invalidates the baseline. The same request can tokenize differently under another model. Recount representative requests and reapprove thresholds whenever routing or model versions change.

Error-path operator workflow

  1. Stop the outbound generation request if the counter is unavailable, the request exceeds the hard stop, compaction returns no usable state, or a quality anchor fails.
  2. Assign one sanitized error class: counter_unavailable, threshold_exceeded, compaction_failed, post_count_high, or quality_anchor_failed.
  3. Do not retry compaction automatically more than once for the same turn.
  4. Apply one approved fallback: remove nonessential replayable context, start a fresh conversation with a user-approved carry-forward summary, route to a model with an appropriate verified context contract, or request human review.
  5. Count the fallback request from scratch and rerun every quality anchor.
  6. Resume only when the request is below the hard stop and quality checks pass. Otherwise, return a clear limitation to the application instead of silently discarding context.
  7. Log the decision, token measurements, policy version, and result without logging conversation content.

FAQ

Should every long conversation be compacted?

No. A conversation should be compacted when its measured request crosses a workload threshold and expected future reuse justifies the operation. A conversation near completion may cost less to finish unchanged than to compact and validate.

Is message count a useful trigger?

Only as a rough warning. Messages vary widely in size, and tools, documents, images, system instructions, and provider state can contribute tokens. The primary trigger should be the full request count for the selected model.

Can a preflight count be trusted exactly?

Treat it according to the provider contract. Anthropic labels its count an estimate, while Google separates a pre-request input count from detailed usage returned after an interaction. Store both estimate and actual usage when available, then monitor the variance.

How do I know whether compaction saves money?

Measure the compaction event under current contract rates, then measure the input-token reduction on later turns. Divide compaction cost by expected per-turn savings to estimate break-even reuse. Include quality failures and retries in the observed cost; a smaller request that produces unusable answers is not a saving.

Can I simply delete the oldest messages?

Only if the workload policy permits it and quality anchors still pass. Age does not determine importance. An early instruction or accepted constraint may remain essential after many newer turns.

Does caching replace compaction?

No. A cache checkpoint can reduce repeated processing costs or latency where the provider supports it, while compaction reduces the amount of state carried forward. Verify both mechanisms independently and avoid assuming cached context no longer consumes context-window capacity.

What if the provider has no compaction API?

Use a controlled client-side policy: identify required state categories, create a bounded carry-forward representation, count the reconstructed request, and compare its behavior with an uncompacted baseline. Stop if the representation cannot preserve the approved quality anchors.

Reader next step

Choose one high-volume multi-turn workload and measure a representative sample before changing behavior. Capture preflight input, actual input, output, tool or cached categories when available, turn number, latency, and outcome. Use the sample to define a warning threshold, compaction threshold, hard stop, and five to ten quality anchors.

Run the policy in observation mode first. It should calculate what it would have done without changing conversation state. Review false triggers, estimator variance, and the projected break-even point. Then enable compaction for a limited workload share with an immediate rollback path.

Use Count CometAPI tokens before budget approval to establish the preflight measurement step. Pair it with Token usage evidence for CometAPI budget reviews so estimates and actual usage reach the same review record. For agentic workloads, also compare the history policy with per-run agent turn guardrails so context growth and loop growth are controlled separately.