TopolearnToken bills grow quietly with traffic; a handful of design choices often cuts them several-fold.
Cost is tokens times price, and price depends on the model and the processing tier. Latency (how long users wait) depends mostly on the model, the reasoning effort and how many tokens are generated. The main levers:
- Right-size the model. Send simple steps (classification, extraction, routing) to an efficient model and reserve the most capable one for hard work. A cheap first pass deciding which model to use is a common pattern.
- Reasoning effort. Hidden reasoning tokens are billed as output. Lower effort on easy tasks is often the biggest single saving.
- Output length. Output costs more than input. Ask for concise formats, use structured outputs, and set
max_output_tokenswith room for reasoning. - Prompt caching. Keep long, stable prefixes identical so they are served from cache.
- Batch and flex. Work that can wait costs about half through the Batch API or
service_tier="flex". - Context size. Trim or compact long conversations; every past turn is billed again on every request, even with
previous_response_id. - Built-in tool usage. Web search, file search and containers have their own charges; limit them where you can.
Measure, don't guess: log usage (input, cached, output and reasoning tokens) with the model and a feature name on every call, and add it up per feature. Set project budgets and alerts in the dashboard. For latency, stream responses, keep prompts short, and use smaller models or lower effort where quality allows.
Example
from openai import OpenAI
client = OpenAI()
LOG = []
def ask(feature: str, text: str, hard: bool = False):
model = "gpt-6-astra" if hard else "gpt-6-luna"
r = client.responses.create(
model=model, input=text, max_output_tokens=2000,
reasoning={"effort": "medium" if hard else "low"},
)
u = r.usage
LOG.append({"feature": feature, "model": model, "in": u.input_tokens,
"cached": u.input_tokens_details.cached_tokens,
"out": u.output_tokens,
"reasoning": u.output_tokens_details.reasoning_tokens})
return r.output_text
ask("tagging", "Tag this review: 'Great battery life'")
ask("planning", "Plan a 3-city rail trip under 500 EUR", hard=True)
for row in LOG:
print(row)Remember
- Biggest levers: model routing, reasoning effort, output length
- Caching, Batch and flex cut price for the right workloads
- Log usage per feature and set budgets
Watch out for
- Watching input tokens while hidden reasoning tokens make up most of the bill.