Keep CometAPI CI Tests from Making Paid Model Calls

Last reviewed: November 6, 2026 (Asia/Shanghai).

Direct answer

Keep routine CometAPI CI tests offline by passing a test-owned HTTP client into the adapter, answering requests through a mock transport, and rejecting unexpected network access. Give that job no usable model credential. Run deliberate live checks in a separate job with its own attempt allowance, output limit, and cost reconciliation.

HTTPX’s transport documentation explicitly supports returning predetermined responses through MockTransport instead of making network requests. That lets a test exercise request construction, response parsing, and local error handling without purchasing another generation on every run. Add a network guard because a mock attached to one client cannot control a second client created elsewhere.

Who this is for

This guide is for Python service owners and AI budget owners whose pull-request tests exercise a CometAPI adapter. It covers non-streaming text requests and ordinary application tests. Model-quality evaluations still need representative generated answers and their own approved plan; the site’s evaluation-budget guide covers that decision.

The example below was checked locally with synthetic responses. No authenticated CometAPI request or billing experiment was performed. The CI design and acceptance criteria are recommendations, and the cost example uses hypothetical prices.

Key takeaways

• Default routine tests to synthetic responses and absent live credentials.
• Inject the client used by the application; test the real adapter through that boundary.
• Fail unexpected requests instead of forwarding them to a real transport.
• Use socket blocking as a diagnostic guard, with runner-level controls for broader coverage.
• Keep synthetic token usage out of actual-spend records.
• Price live checks across the entire CI matrix, including retries and reruns.

Contract details to verify

Decide what each job is allowed to prove

Write the job’s permission before selecting its tests:

A fixture returning OK proves the application’s handling of that fixture. It cannot establish current model availability, answer quality, or final billed cost. Conversely, a successful live generation does not exercise every malformed-body branch in your parser. Retain both kinds of test with clear ownership.

CometAPI’s chat-completion reference documents POST /v1/chat/completions, model and message fields, completion choices, finish reasons, and usage. Build fixtures from the contract your adapter supports. For a live run, verify the actual model ID and its output-limit parameter against current documentation; fixture-model below is only a local label.

Make the offline path explicit

Construct the client at an application composition boundary and pass it to the adapter. Avoid a module import that immediately constructs a live SDK client or looks up production credentials. An offline test should not need a real key to import the code it tests.

Create the mock unconditionally in the test fixture. A missing environment variable should fail setup rather than select a real transport. HTTPX’s documentation illustrates switching transports with an environment variable; the stricter recommendation here makes the ordinary test path explicit. Likewise, an unrecognized request should raise a test failure. Do not add a fallback that sends unmatched requests upstream.

Keep fixture bodies synthetic and small. Include the fields the adapter consumes, plus deliberate failure cases. Mark synthetic usage as fixture data in any test ledger or event sink. A fabricated usage object is useful for checking arithmetic and parsing, but it supplies no evidence of an actual provider charge.

Smoke-test workflow

Exercise the adapter without a network transport

Save this example as tests/unit/test_ci_boundary.py. It includes a deliberately small adapter function; replace that function with your application’s adapter while retaining client injection and the assertions. A production adapter also needs its own completion, schema, and usage-validation rules.

import json
import socket

import httpx
import pytest
from pytest_socket import SocketBlockedError


def generate(client, model):
    response = client.post(
        "/v1/chat/completions",
        json={
            "model": model,
            "messages": [{"role": "user", "content": "Reply with OK."}],
            "max_completion_tokens": 64,
            "stream": False,
        },
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]


def test_request_contract():
    calls = []

    def handler(request):
        assert request.method == "POST"
        assert str(request.url) == "https://ci.invalid/v1/chat/completions"
        body = json.loads(request.content)
        assert body == {
            "model": "fixture-model",
            "messages": [{"role": "user", "content": "Reply with OK."}],
            "max_completion_tokens": 64,
            "stream": False,
        }
        calls.append(body)
        return httpx.Response(
            200,
            json={
                "id": "chatcmpl-ci-fixture",
                "choices": [{
                    "index": 0,
                    "message": {"role": "assistant", "content": "OK"},
                    "finish_reason": "stop",
                }],
                "usage": {
                    "prompt_tokens": 4,
                    "completion_tokens": 1,
                    "total_tokens": 5,
                },
            },
        )

    with httpx.Client(
        base_url="https://ci.invalid",
        transport=httpx.MockTransport(handler),
        trust_env=False,
    ) as client:
        assert generate(client, "fixture-model") == "OK"
    assert len(calls) == 1


def test_python_socket_guard_is_active():
    with pytest.raises(SocketBlockedError):
        socket.socket(socket.AF_INET, socket.SOCK_STREAM)


def test_http_error_does_not_trigger_another_attempt():
    calls = []

    def handler(request):
        calls.append(request)
        return httpx.Response(429, json={"error": {"message": "fixture limit"}})

    with httpx.Client(
        base_url="https://ci.invalid",
        transport=httpx.MockTransport(handler),
        trust_env=False,
    ) as client:
        with pytest.raises(httpx.HTTPStatusError):
            generate(client, "fixture-model")
    assert len(calls) == 1

Run it with the socket plugin installed:

python -m pytest -q --disable-socket tests/unit/test_ci_boundary.py

Local verification used Python 3.12.3, HTTPX 0.28.1, pytest 9.2.0, and pytest-socket 0.8.1. The three tests passed with one expected warning from the blocked socket probe. The HTTP 429 test confirmed one handler invocation for this adapter, which contains no retry loop. That observation does not establish the retry behavior of a different SDK or production wrapper.

The socket probe creates no connection. If the guard is absent or overridden, the constructor succeeds and the test fails because the expected exception is missing. The happy-path test checks exact request fields and one synthetic response. Its mock client omits authentication entirely and uses a placeholder host; live authentication belongs in the separate integration check.

Check the guard’s scope

The upstream pytest-socket documentation describes --disable-socket, per-test exceptions, host allowances, and a --force-enable-socket override. Review configuration and markers so ordinary tests cannot quietly regain network access. Where async tests require Unix-domain sockets, the project documents --allow-unix-socket; verify that allowance against the test’s needs.

The same documentation warns about fixture ordering and teardown behavior. A test-function check alone is insufficient evidence for imports, session-scoped setup, subprocesses, native libraries, or another runner process. Treat the plugin as a Python test guard. For broader prevention, configure the test execution environment to deny external inference access, omit usable credentials, and check those controls independently. Keep dependency installation in a separate preparation step when the test environment requires outbound access to obtain packages.

Do not report the whole job as network-free merely because the two mocked HTTP requests passed. Audit other client constructors, test setup, background tasks, subprocesses, and teardown. Include an acceptance check for accidental creation of an unmocked client in your actual application path.

Budget the deliberate live checks

Count the complete CI fan-out before approving paid checks. A hypothetical 120 workflow runs per day, six matrix jobs per run, and 40 model attempts per job produce 28,800 attempts. At an assumed average of $0.002 per attempt, that is $57.60 per day before extra reruns, retries, or judges. These are illustrative planning inputs, not CometAPI prices or measured consumption.

Moving deterministic adapter assertions to fixtures removes their need for live generations. It does not replace the evidence collected by quality evaluations. Schedule the remaining integration checks according to the behavior that needs live verification, then forecast each call class separately:

forecast_model_cost = sum(planned_attempts_c * estimated_cost_per_attempt_c)

Use the exact selected route’s pricing and billing units. CometAPI’s pre-call cost guide distinguishes estimates from actual usage and billing records, and illustrates output limits. An average-cost forecast is not a hard admission bound. Apply verified input and output limits, count every permitted retry, and exclude request classes whose exposure the approved policy cannot cover.

Give the live job a dedicated credential and an explicit owner, expiry, model, attempt ceiling, output limit, and price revision. Enforce one shared allowance across parallel shards, or allocate disjoint allowances whose sum fits the approved total. The site’s atomic reservation workflow explains financial admission before dispatch.

A rerun needs the same accounting discipline as the first run. Record every live attempt against its workflow run, job, shard, and attempt identity. After dispatch, a timeout leaves cost unresolved until usage or billing evidence settles it. A passed offline suite supplies no reason to release that live commitment.

Failure modes

The test mocks a client the application never uses. A helper constructs a second client and bypasses the fixture. Inject the client at the actual adapter boundary and make unexpected network access fail visibly.

Missing fixture data causes a live fallback. A permissive handler forwards an unfamiliar request to the normal transport. Reject unmatched method, path, and body combinations instead.

A test credential is also a production credential. Removing a variable from the test function leaves a usable key in runner configuration. Remove live credentials from the offline job’s environment and secret-resolution paths.

The plugin is treated as a runner firewall. Setup order, exceptions, or another process create a path beyond the demonstrated guard. Verify the execution environment separately and report the guard’s actual scope.

Fixture usage becomes real spend. A test emits its invented token totals into a shared accounting stream. Use a test-owned sink and explicit synthetic provenance; keep actual-spend reconciliation tied to real attempts.

Each shard receives the whole live budget. Parallel jobs independently admit work against the same nominal amount. Use shared admission or bounded allocations before any shard dispatches.

Sources checked

Four primary public references were fetched on November 6, 2026:

• HTTPX: Transports — client transport injection, mock handlers, and predetermined responses without network requests.
• pytest-socket: upstream README — socket blocking, explicit overrides, allowances, fixture-order limitations, and teardown behavior. Check these against the installed plugin release.
• CometAPI: Create a chat completion — endpoint, request fields, completion structure, and usage.
• CometAPI: Estimate request cost before calling a model — route-specific units, output budgeting, and estimate-versus-actual reconciliation.

These maintained technical references support a current operating task rather than a dated announcement. The local test result applies to the versions stated above; recheck library and model contracts when upgrading.

FAQ

Does passing these tests prove the current model works?

It proves local handling of the supplied fixtures. Use a separately authorized live check for authentication and interface observations, and a declared evaluation plan for answer quality.

Should we delete live integration tests?

Retain live tests that answer a defined question. Move deterministic request, parser, and error assertions into offline tests, then give the remaining live work its own schedule and budget.

Does zero live model traffic make CI free?

The runner and other services can still incur costs. This control addresses accidental model dispatch from the covered test job; it does not establish a zero-cost CI platform.

Reader next step

Choose one adapter test that currently makes a live call. Inject its HTTP client, replace the response with synthetic data, and run the request, error, and socket-guard checks. Verify that the surrounding test job has no usable live credential and that unexpected network activity fails.

Then define the smallest remaining live check and its allowance. When that job is ready, Start with CometAPI to verify the approved route with bounded, non-sensitive input. Reconcile its actual attempts and usage before expanding the schedule.