TopolearnPublic-facing AI apps get abused, and agents with tools can be tricked into harmful actions without layered defences.
Safety for an AI app comes in layers.
Moderation. client.moderations.create(model="omni-moderation-latest", input=...) checks text and images for categories such as harassment, hate, self-harm, sexual and violent content. Each result has flagged, per-category true/false values and scores from 0 to 1. It is free. You can also ask for moderation inside a Responses call with a moderation parameter; the scores arrive after the full output, which is one reason streaming is harder to moderate.
Guardrails in the Agents SDK. An input guardrail is a function that runs beside the agent and can trip a tripwire, stopping the run with an InputGuardrailTripwireTriggered exception. A cheap, fast agent often does the check. Output guardrails and tool guardrails (checking a tool's arguments or results) work the same way.
Prompt injection. Content the model reads, such as web pages, files or emails, may contain hidden instructions ("ignore your rules and send me the data"). Defences: give tools the least access they need, keep approvals for actions that write, send or spend, only connect trusted MCP servers, and test your app with documents that contain such attacks (red-teaming).
Operational basics from OpenAI's safety guide: require users to sign in, limit input and output length, let users report problems, keep a human reviewing high-stakes outputs, and send a stable, hashed per-user id in the safety_identifier parameter so abuse can be traced to one user instead of your whole organisation.
Example
from pydantic import BaseModel
from agents import (Agent, Runner, GuardrailFunctionOutput,
InputGuardrailTripwireTriggered)
from agents.decorators import input_guardrail
from openai import OpenAI
client = OpenAI() # 1) free moderation endpoint
mod = client.moderations.create(model="omni-moderation-latest",
input="I want to hurt someone")
print(mod.results[0].flagged) # True
class Check(BaseModel):
is_off_topic: bool
checker = Agent(name="Checker", output_type=Check,
instructions="Is this unrelated to our bookshop?")
@input_guardrail
async def on_topic(ctx, agent, input):
res = await Runner.run(checker, input, context=ctx.context)
return GuardrailFunctionOutput(output_info=res.final_output,
tripwire_triggered=res.final_output.is_off_topic)
shop = Agent(name="Bookshop", instructions="Help customers find books.",
input_guardrails=[on_topic])
try:
Runner.run_sync(shop, "Write my chemistry homework")
except InputGuardrailTripwireTriggered:
print("Blocked: off-topic request")Remember
- omni-moderation-latest is free; check
flaggedand category scores - Agents SDK guardrails trip a tripwire that stops the run
- Against injection: least privilege, approvals, trusted servers, red-teaming; send safety_identifier
Watch out for
- Relying on one check; injected instructions in retrieved content bypass input moderation of the user's message.