TopolearnBoth cut cost with little code: one by ordering your prompt well, one by trading speed for price.
Prompt caching is on by default. When a request starts with exactly the same text as a recent one (the same prefix), OpenAI reuses the work already done for it, which makes those input tokens cheaper and the first token arrive sooner. On current models the prefix must be at least 1,024 tokens.
You get the benefit by structuring prompts well:
- Put stable content first: instructions, tool definitions, examples, long reference documents.
- Put changing content last: the user's question, today's date, per-user data.
- Keep the stable part byte-for-byte identical. One changed character early on breaks the match for everything after it.
Check usage.input_tokens_details.cached_tokens to see how many input tokens were served from cache. An optional prompt_cache_key groups related requests (for example per customer). Newer models also offer settings to control caching explicitly; see the caching guide for the model you use.
Flex processing is a cheaper tier for work that is not urgent: set service_tier="flex". Tokens are billed at Batch API rates, but responses are slower and occasionally the request fails with a 429 "resource unavailable" (not charged). Raise the client timeout, and on failure either retry later with backoff or fall back to the normal tier with service_tier="auto". Flex is in beta and only some models support it.
Good fits for flex: evaluations, data enrichment and background jobs that still need the regular Responses API.
Example
import openai
from openai import OpenAI
client = OpenAI(timeout=900.0) # flex can be slow
STABLE = open("style_guide.txt").read() # long, identical every call
def review(text: str):
kwargs = dict(model="gpt-6-astra", instructions=STABLE,
input=f"Review this paragraph:\n{text}") # varying part last
try:
return client.responses.create(service_tier="flex", **kwargs)
except openai.RateLimitError: # flex capacity unavailable
return client.responses.create(service_tier="auto", **kwargs)
r = review("Our product are great.")
print(r.usage.input_tokens, r.usage.input_tokens_details.cached_tokens)Remember
- Caching is automatic: stable prefix first, varying content last
- Measure with usage.input_tokens_details.cached_tokens
- service_tier="flex": batch-level price, slower, may 429
Watch out for
- Putting a timestamp or user id at the top of the instructions, which defeats caching.