TopolearnBulk work that can wait is half the cost and has its own rate limits, so it doesn't compete with live traffic.
Some jobs do not need an instant answer: classifying a product catalogue overnight, embedding a document archive, running an eval. The Batch API processes many requests in the background at about 50 percent of the normal price.
The flow uses files:
- Write a JSONL file, a text file with one JSON object per line. Each line has a
custom_id(your label),method: "POST", the endpointurl(such as/v1/responsesor/v1/embeddings) and abodywith the usual request parameters. One file targets one model. - Upload it with
client.files.create(file=..., purpose="batch"). - Start the job with
client.batches.create(input_file_id=..., endpoint=..., completion_window="24h").24his currently the only window. - Poll
client.batches.retrieve(id)untilstatusiscompleted(other statuses includevalidating,in_progress,failedandexpired). - Download
output_file_idwithclient.files.content(...). Failed lines go toerror_file_id.
Output lines can be in any order, so match them by custom_id. A batch can hold up to 50,000 requests and a 200 MB input file. Batch has its own queue limits, separate from normal per-minute limits, so it won't slow down your live app. Output files are deleted 30 days after completion; save what you need.
Example
import json, time
from openai import OpenAI
client = OpenAI()
reviews = {"r1": "Loved it!", "r2": "Broke after a day."}
with open("batch.jsonl", "w") as f:
for rid, text in reviews.items():
f.write(json.dumps({"custom_id": rid, "method": "POST", "url": "/v1/responses",
"body": {"model": "gpt-6-luna",
"input": f"Sentiment in one word: {text}"}}) + "\n")
up = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=up.id, endpoint="/v1/responses",
completion_window="24h")
while (batch := client.batches.retrieve(batch.id)).status not in (
"completed", "failed", "expired", "cancelled"):
time.sleep(60)
if batch.output_file_id:
for line in client.files.content(batch.output_file_id).text.splitlines():
row = json.loads(line)
print(row["custom_id"], row["response"]["status_code"])Remember
- JSONL in, JSONL out; about half price; 24-hour window
- Each line: custom_id, method, url, body
- Match results by custom_id; save outputs before they expire
Watch out for
- Mixing two models in one input file; each batch file targets a single model.