TopolearnThe same feature can cost ten times more or feel ten times slower depending on these choices, often with no quality gain.
Three things trade off: quality, cost (mostly tokens times price per token) and latency (how long the user waits: until the first words appear, and until the answer is complete).
Choosing a model:
- Providers offer tiers: large, most capable models; mid-size all-rounders; and small, fast, cheap ones (and open models you can host yourself). Small models are often good enough for classification, extraction and routing.
- Pick per task, not per product. A router can send easy requests to a small model and hard ones to a large one.
- Decide with your evals, not leaderboards: the cheapest model that passes your tests wins. Re-check when new models appear, which is often.
Levers within a model:
- Reasoning effort. Thinking tokens cost money and time; use low effort for simple steps.
- Output length. Output tokens are slower and usually pricier than input; ask for concise answers and cap the length.
- Input length. Send the relevant context, not everything. Long histories and big tool results are a common hidden cost in agents.
- Prompt caching. If many requests start with the same long text (system prompt, tool definitions, a reference document), providers can cache it so repeat input is cheaper and faster. Put the stable part first and the changing part last.
- Batch processing. For work that can wait (overnight tagging of thousands of records), batch interfaces cost less in exchange for slower turnaround.
- Streaming. Showing tokens as they arrive does not reduce total time but makes the wait feel much shorter.
Track tokens per request and per task in production; costs usually surprise in agents, where one task can make dozens of calls.
Example
# Estimate monthly cost for one feature. Fill in prices from your provider's pricing page.
PRICE_IN_PER_M = 0.0 # price per 1 million input tokens
PRICE_OUT_PER_M = 0.0 # price per 1 million output tokens (reasoning tokens count here)
requests_per_day = 2_000
input_tokens = 1_500 # system prompt + context + question
output_tokens = 300 # answer (+ any reasoning tokens)
monthly_in = requests_per_day * 30 * input_tokens
monthly_out = requests_per_day * 30 * output_tokens
cost = monthly_in / 1e6 * PRICE_IN_PER_M + monthly_out / 1e6 * PRICE_OUT_PER_M
print(f"{monthly_in:,} input + {monthly_out:,} output tokens per month -> ${cost:,.2f}")
# 90,000,000 input + 18,000,000 output tokens per month -> $...Remember
- Trade quality, cost and latency per task; route easy work to small models
- Let evals choose the cheapest model that passes
- Levers: reasoning effort, shorter input and output, prompt caching, batching, streaming