How LLM applications actually work with the Responses API
Trace context, instruction authority, streaming, structured outputs, tools, retries, routing, and token cost through one LLM request.
The whole story in 7 lines
A dependable LLM product is a model inside a carefully budgeted, validated, observable, and permissioned software loop.
- Instructions, history, user input, tools, and output reserve all compete for one finite context budget.
- Outside content remains untrusted even when its sentences look like commands.
- Generation chooses tokens in order, while streaming exposes typed partial events before completion.
- Strict schemas guarantee the required shape, while valid JSON alone can still violate the application contract.
- The model proposes a tool call, but application code validates, executes, and returns the result.
- Routing picks an appropriate path, while bounded retries recover only from failures worth retrying.
- Token usage becomes cost, so trimming repeated context can matter as much as shortening the final answer.
The hidden system
A customer asks where order 1042 is, yet one model response must cross several software boundaries before the product can trust it. Our opening map reveals those hidden jobs.
Context is the model workspace, while an instruction tells it how to behave. A schema defines the output shape, and a tool connects the request to outside work.
The journey starts by packing a request, then separates trusted rules from untrusted content. After generation begins, the application streams, validates, executes, recovers, and measures.
Every stop supports the same order request, so outputs from one stage become inputs to the next. Now let us begin with the finite workspace inside every model call.
Request and context
We begin with one blue request tray that represents the model context. Stable instructions, earlier conversation, user input, and tool definitions all need space inside it.
The application also reserves capacity for generated output because input and output share the model window. Filling every slot with history can leave no room for a complete answer.
Our full history pushes the combined request beyond the illustrative sixty four token window. Which segment can shrink without removing the current goal or its tool contract?
The overflow edge now makes the failure concrete. A model cannot squeeze extra tokens into a fixed context, so the application must trim, summarize, or reject before sending.
Switch the History policy control through both options. Compare the downstream result badge and finish when you have exposed both the overflow and the request that fits.
Context is a deliberate request budget, not unlimited memory. Once the packet fits with output room intact, we can ask which parts are trusted to direct behavior.
Authority boundaries
★ If you remember one thing · Readable outside text remains untrusted data and cannot become application policy merely by sounding like an instruction.
Now that the request fits, we must separate instruction authority from ordinary content. The upper stack holds application policy and the user goal, while retrieved material enters below a trust boundary.
The retrieved document can help answer the question, but it may also contain attacker written commands. OpenAI describes prompt injection as malicious instructions hidden inside content the model may encounter.
The document now demands every customer record and tries to override the developer rule. Since the model can read those words, should the application let them reach a sensitive tool?
The unsafe path stops below the red boundary while the trusted purple stack continues toward the allowed order lookup. This contrast is the core security idea around prompt injection.
A warning sentence alone cannot create a security boundary. The application must limit available tools, validate arguments, protect private data, and review consequential writes. Next, we follow the allowed request into generation.
Generation and streaming
The trusted request reaches generation, where the model chooses one next token from a probability distribution. That token joins the sequence and becomes part of the context for the following choice.
As generation continues, the Responses API can emit typed server sent events. Common lifecycle events mark creation, text deltas, completion, and errors, so application code can handle each case directly.
Both delivery modes require the same four illustrative chunks to finish. Which mode lets the support interface start helping before the final chunk exists?
The streaming lane now contains an early order update while the batch lane remains closed until completion. The model did not finish sooner, but the product received useful work earlier.
Switch the Delivery mode control through both options. Compare the downstream first update marker, then finish after you have experienced early partial output and waiting for completion.
Streaming is an interface and processing choice with extra moderation considerations for partial text. Next, we turn the finished answer into a contract that application code can safely consume.
Structured outputs
We just received generated text, but the support interface needs three typed fields. A JSON object can parse successfully and still omit eta_days or invent a status outside the allowed choices.
The schema gate turns those expectations into visible slots for order_id, status, and eta_days. Structured Outputs are designed to make model responses adhere to the supplied JSON Schema.
Our JSON mode object is valid JSON, yet one field is missing and the status value is outside the accepted set. Will a parser alone make this support card safe to render?
The strict object fills every required slot with an allowed value, so the green support card can accept it. Refusals still need their own branch because a safety refusal may not match the requested schema.
Switch the Output contract control through both options. Compare the consumer badge and complete the experiment after you expose both the rejected shape and the accepted typed card.
A response contract removes format guesswork, but it does not prove every value is factually correct. Next, the application will fetch the real order record through a tool.
The tool loop
The schema tells us what answer shape we want, but the model still needs current order data. The application sends a tool definition with the request, describing get_order and its typed arguments.
The model returns a function call for order 1042 instead of inventing a status. This item expresses what the model wants to call, while the real database remains outside the model.
The proposed arguments now sit at the application boundary. Who must validate those arguments and execute the order lookup?
Application code validates order_id, runs the lookup, and returns the green tool output along the lower path. The model receives that result in a second request and produces a grounded answer.
The loop may continue when another tool call is needed, but permissions and side effects still belong to application policy. That boundary remains across every repeated model call. Next, we design the routing and retry behavior around this loop.
Routing and retries
The tool loop works, but production traffic has different difficulty and failure modes. A routing score sends ordinary questions down a fast path while harder requests use the deeper path.
A failure then enters an error classifier before any retry occurs. Rate limits may clear with time, while invalid credentials need configuration repair rather than another identical request.
Our retryable request has three remaining attempts and one hard deadline. Should those attempts fire together or spread out with increasing waits?
The amber waits grow across the retry ladder, yet the last attempt remains before the red deadline. A permanent authentication error takes the short stop path instead of wasting the same budget.
Switch the Failure kind control through both options. Compare the downstream decision and finish after you have exposed a bounded retry path and an immediate stop.
Reliable orchestration retries only errors worth retrying and always stops within a known budget. With the path stable, we can measure the tokens and cost carried by each request.
Token economics
The reliable request now produces usage counts for input and output tokens. The application converts each count with its model rate, then multiplies the per request amount by traffic.
Conversation state affects this math because prior context may be included and billed again on later responses. OpenAI notes that previous input tokens in a response chain still count as input tokens.
The newest user message is short, but a large prior history sits underneath it. Which policy prevents that repeated prefix from growing with every follow up?
The trimmed bar removes the large repeated prefix before the same illustrative rates and daily volume are applied. The result is a smaller cost area without changing the validated order answer.
Switch the State policy control through both options. Compare the downstream cost badge and finish after you expose both growing input cost and bounded input cost.
Token accounting connects architecture choices to an operating budget, so teams should measure real usage instead of guessing from prompt length. Now let us step back and see the whole picture together.
The complete system
We started by packing instructions, history, user input, tools, and output reserve into one finite context budget. Good requests leave deliberate room for generation.
Then we separated trusted application policy from untrusted outside content. The model may read retrieved text, but software boundaries constrain what that text can authorize.
Next, generation chose tokens in sequence while streaming exposed typed partial events. That gave the interface an earlier useful update without changing the final completion time.
Structured Outputs then matched the answer to a JSON Schema, which let the support card consume required fields and handle refusals through a separate branch.
The tool loop showed that the model proposes calls while application code validates, executes, and returns results. That boundary keeps permissions and side effects inspectable.
Routing selected an appropriate path, and the failure classifier retried only errors that might clear. Increasing waits, jitter, attempt limits, and deadlines kept recovery bounded.
Finally, usage converted architecture into cost. The numbers in this lesson were illustrative, while the API mechanics came from official OpenAI documentation checked for this rebuild.
All seven spokes now support one green response. A production LLM application is dependable when ordinary software makes the model call budgeted, permissioned, validated, recoverable, observable, and affordable.
The whole story in 7 lines
A dependable LLM product is a model inside a carefully budgeted, validated, observable, and permissioned software loop.
- Instructions, history, user input, tools, and output reserve all compete for one finite context budget.
- Outside content remains untrusted even when its sentences look like commands.
- Generation chooses tokens in order, while streaming exposes typed partial events before completion.
- Strict schemas guarantee the required shape, while valid JSON alone can still violate the application contract.
- The model proposes a tool call, but application code validates, executes, and returns the result.
- Routing picks an appropriate path, while bounded retries recover only from failures worth retrying.
- Token usage becomes cost, so trimming repeated context can matter as much as shortening the final answer.