TopolearnAt scale, limits shape your design: queues, batching and caching all exist partly because of them.
OpenAI limits how much each organisation and project can send. Limits are measured in several ways, and whichever fills first stops you:
- RPM / RPD: requests per minute or per day.
- TPM / TPD: tokens per minute or per day.
- Others for images and audio.
Limits differ per model, and some model families share one. They rise with your usage tier, which goes up automatically as your total paid usage grows. The limits page in the dashboard shows current values.
Every response carries headers that tell you where you stand: x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens and matching -reset- headers. Read them with client.responses.with_raw_response.create(...) if you want to slow down before you hit the wall.
When you do get a 429 or a 503:
- Respect
Retry-Afterwhen present, as a minimum wait. - Otherwise use exponential backoff with jitter, a small random extra delay, so many clients don't retry at the same instant.
- Cap both the number of attempts and the total time.
- Failed requests still count against your per-minute limit, so hammering makes it worse.
The SDK already does basic retries. For steady high volume, add a queue that limits concurrency, move non-urgent work to Batch or flex, and cache prompt prefixes. When growing traffic sharply, ramp up gradually.
Example
import random, time
import openai
from openai import OpenAI
client = OpenAI(max_retries=0) # we handle retries ourselves here
def create_with_backoff(max_attempts=5, **kwargs):
for attempt in range(max_attempts):
try:
raw = client.responses.with_raw_response.create(**kwargs)
print("tokens left:", raw.headers.get("x-ratelimit-remaining-tokens"))
return raw.parse()
except (openai.RateLimitError, openai.InternalServerError) as e:
wait = float(e.response.headers.get("retry-after") or 2 ** attempt)
time.sleep(wait + random.uniform(0, 1)) # jitter
raise RuntimeError("gave up after retries")
r = create_with_backoff(model="gpt-6-luna", input="ping")Remember
- Limits: requests and tokens per minute/day, per model, by tier
- Read x-ratelimit-* headers; honour Retry-After
- Backoff with jitter and a cap; shift bulk work to Batch or flex