15. LLM and Agentic Systems

An LLM application produces model output. An agentic system additionally lets a model select steps, use tools, observe results, and continue until a goal or stopping condition is reached. A workflow fixes most of that control flow in code; an agent chooses more of it at runtime. The boundary is a design decision, not a product label.

This chapter is a current engineering snapshot, not a promise that every model, framework, or protocol will remain unchanged. Provider features and emerging agent protocols move quickly; keep their versions and assumptions explicit.

Evidence: S61, S62, S63, S64, S65

15.1 Start with the least autonomy that works

Use the lowest level that satisfies the requirement:

1. Deterministic function: rules and ordinary code fully define the result.
2. Single model call: one bounded inference, preferably with a structured output contract.
3. Deterministic workflow with model steps: code owns routing, retries, limits, and completion.
4. Single agent: the model chooses among approved tools inside a bounded loop.
5. Multi-agent system: specialized agents coordinate or delegate because the work is genuinely separable.

Move upward only when evaluation shows a material quality or coverage gain. Every increase in autonomy also increases the space of possible trajectories, tool calls, cost, latency, and failure modes. Multi-agent designs are most useful for independent, parallelizable work; shared mutable state and tightly coupled steps often make them slower and harder to debug.

Evidence: S63, S65, S69

15.2 Define the agent contract before choosing a framework

For each agent, write down:

• Goal and user value.
• Non-goals and prohibited outcomes.
• Inputs, output schema, and evidence/citation expectations.
• Allowed data sources and their trust level.
• Allowed tools, permissions, and side effects.
• Autonomy boundary: what it may decide, propose, or execute.
• Human approval points and who can approve.
• Time, token, tool-call, recursion, and spend budgets.
• Completion, abstention, escalation, and cancellation conditions.
• Quality, safety, latency, reliability, and cost targets.
• State retention, privacy, deletion, and tenant-isolation rules.

“Be helpful and safe” is not a usable system specification. Bind behavior to observable acceptance conditions and executable evaluations.

Evidence: S46, S53, S63, S68, S85, S92

15.3 Reference architecture

The model is one component inside a larger deterministic control plane. The control plane should enforce identity, authorization, policy, budgets, approvals, state transitions, and auditability even when model output is wrong.

Figure 7. A bounded agent loop inside a deterministic control plane.

Evidence: S63, S64, S84, S89, S90

15.4 Model layer

Choose a model against the actual task and evaluation set, not a leaderboard alone. Record the exact model/provider version or alias, capabilities, context limit, supported tool/structured-output features, latency, price, data-handling terms, and regional availability.

Design for these properties:

• Probabilistic output: identical inputs can produce different results. Test multiple trials.
• Capability variation: reasoning, multilingual quality, tool use, vision, and long-context behavior differ by model and release.
• Version drift: aliases, defaults, and safety behavior can change. Pin versions where supported and run regression evaluations before migration.
• Fallback semantics: a cheaper or alternate model may not satisfy the same contract. Validate each route separately.
• Output validation: parse and validate structured output; treat free text as untrusted data.
• Abstention: define when the system must say it cannot complete the task safely or reliably.

Use model routing only when the quality/cost benefit exceeds the operational complexity. Do not silently route sensitive data to a provider or region that violates the data contract.

Evidence: S59, S63, S68, S85, S86

15.5 Instructions and context engineering

An agent sees only the context assembled for the current step. Context can contain system and developer instructions, user input, conversation state, retrieved documents, tool descriptions, tool results, and memory. More context is not automatically better: irrelevant or conflicting material consumes attention and can reduce reliability.

Design the context builder to:

• Preserve an explicit instruction hierarchy and scope.
• Keep untrusted content labeled as data, never as authority.
• Retrieve only the sources needed for the next decision.
• Include provenance, access-control result, freshness, and stable identifiers.
• Summarize or compact long histories while retaining decisions, open work, and safety constraints.
• Place key constraints where they remain visible during long runs.
• Measure context size, cache behavior, truncation, and retrieval quality.
• Make source conflict and missing evidence visible to the model and user.

Prompt injection cannot be solved by a stronger prompt alone. Enforce authorization and side-effect policy outside the model, minimize privileges, validate tool requests, and isolate untrusted execution.

Evidence: S66, S81, S82, S83, S84, S85

15.6 Retrieval-augmented generation and knowledge

RAG combines model generation with retrieved external knowledge. It can improve freshness, provenance, and domain coverage, but retrieval creates its own correctness and security boundary.

Specify the full retrieval path:

1. Source ingestion, ownership, classification, and deletion.
2. Parsing, chunking, metadata, and document versioning.
3. Index/embedding model and rebuild strategy.
4. Query transformation and permission-aware candidate retrieval.
5. Filtering, reranking, deduplication, and context packing.
6. Citation mapping back to immutable source identifiers.
7. Evaluation for recall, precision, answer grounding, and access-control leakage.

Never let an index bypass source permissions. Treat retrieved text as untrusted, preserve tenant boundaries, and make stale/deleted-content handling explicit. For exact identifiers, policies, or numbers, combine semantic retrieval with deterministic lookup where appropriate.

Evidence: S60, S66, S81, S82

15.7 Tools and the action plane

A tool is an API exposed to the model through a name, description, and input/output contract. Tool quality often matters more than adding prompt instructions around a weak tool.

For every tool define:

Prefer narrow, composable tools over a generic shell or unrestricted API. Separate read tools from write tools. Normalize large results before returning them to the model. Validate authorization again inside the tool; never trust the model’s assertion that permission exists.

Evidence: S62, S64, S67, S73, S74, S82, S83, S89

15.8 State, memory, and durable execution

Do not call every stored value “memory.” Distinguish:

• Run state: current step, pending tool call, budgets, approvals, and checkpoints.
• Conversation state: messages or a compact representation needed for continuity.
• Working memory: temporary notes, plans, and intermediate artifacts for a task.
• User memory: durable preferences or facts saved with user awareness and control.
• Semantic knowledge: indexed documents or records retrieved from an authoritative source.
• Audit history: immutable operational evidence with a defined retention policy.

Long-running agents need checkpointed state, resumable approvals, idempotent tool execution, cancellation, leases/ownership, and recovery after process failure. Never resume by blindly replaying a side effect. Store the tool-call identity and outcome so the orchestrator can determine whether execution already occurred.

For durable user memory, define consent, scope, source, confidence, freshness, edit/delete behavior, retention, encryption, tenant isolation, and whether the memory may be used for model training. Do not turn transient inference into a permanent fact without an explicit rule.

Evidence: S64, S66, S89, S91, S92

15.9 Orchestration and multi-agent patterns

Two control styles are common:

• Code orchestration: deterministic code owns the graph and invokes models at named steps. It is easier to test, bound, and audit.
• LLM orchestration: a model selects the next agent or tool. It is more flexible but requires tighter budgets, policy, evaluation, and traceability.

Common patterns:

Every delegation should carry a bounded task, relevant context, authority, budget, expected artifact, and return condition. The parent remains accountable for integration. Use a single writer or explicit conflict-resolution strategy when multiple agents touch shared state.

Evidence: S64, S65, S69

15.10 Interoperability specifications and contracts

Use a protocol because it solves an interoperability requirement, not because it is fashionable.

MCP and A2A solve different boundaries: MCP commonly exposes context and capabilities to an agent host, while A2A addresses collaboration between independent agents. They can coexist. The MCP roadmap mentions future work such as agent messaging, but a roadmap is not a released compatibility guarantee.

Evidence: S70, S71, S72, S73, S74, S75, S76, S77, S78, S79

15.11 Security, privacy, and human control

Threat-model the complete system: user, gateway, orchestrator, model/provider, context store, retrieval index, tools, remote agents, code sandbox, observability backend, and human approval UI. Mark every trust boundary and data flow.

Minimum controls:

• Treat prompts, retrieved documents, websites, messages, tool results, and remote-agent responses as potentially hostile.
• Enforce least privilege with short-lived, audience-restricted credentials; do not place reusable secrets in model context.
• Bind authorization to the represented user/tenant and recheck it for every action.
• Use allowlists and sandboxing for code, filesystem, browser, network, and process access.
• Gate high-impact, irreversible, financial, destructive, account-changing, or external-communication actions.
• Show the approver the exact action and validated arguments; approval of an earlier plan is not approval of changed arguments.
• Fail closed if an interrupted/pending tool request cannot be reconstructed safely.
• Scan and constrain outputs before they reach interpreters, databases, shells, browsers, or users.
• Keep a kill switch, cancellation path, credential revocation, and incident-response procedure.
• Minimize sensitive telemetry and control who can inspect prompts, tool arguments, and retrieved data.
• Test prompt injection, privilege escalation, data exfiltration, confused-deputy behavior, excessive agency, denial of service, and supply-chain compromise.

Human-in-the-loop is a risk control, not a substitute for safe design. Approval fatigue, ambiguous UI, missing context, or a non-expert approver can make it ineffective.

Evidence: S27, S75, S76, S81, S82, S83, S84, S89, S92

15.12 Evaluation is the acceptance-test layer

An agent evaluation contains a task, initial state, environment, allowed tools, reference or rubric, grader, and repeated trials. Inspect both the final outcome and the trajectory used to reach it.

Use several grader types where appropriate:

• Deterministic state checks for database, file, API, or environment outcomes.
• Schema and invariant checks for structured output.
• Reference-based checks for known-answer tasks.
• Human rubric for nuanced usefulness or safety.
• Model-based graders only after calibrating them against human judgments and adversarial cases.
• Cost, latency, tool-count, policy, and side-effect checks.

Do not accept the agent’s final text as proof that the task succeeded. Verify external state. Run multiple trials because a single green run hides variance. Maintain separate development and held-out regression sets, include realistic failures and adversarial inputs, and preserve representative traces for diagnosis.

Track at least:

• Task success and critical failure rate.
• Success consistency across repeated trials.
• Tool-selection and argument correctness.
• Unauthorized or unnecessary action rate.
• Grounding/citation correctness and retrieval quality.
• Human escalation and approval rates.
• Latency distribution, tokens, tool calls, and cost per successful task.
• Recovery after timeouts, partial failures, interruption, and resume.

Benchmarks such as GAIA, SWE-bench, and tau-bench are useful examples of environment-based evaluation, but product acceptance must use tasks, policies, tools, and failure modes from the real system.

Evidence: S68, S86, S87, S88

15.13 Observability and production operations

Give every run a stable identifier and record the causal graph: model calls, tool calls, handoffs, guardrail results, approval pauses, retries, state transitions, errors, and final verified outcome. Correlate these with ordinary service traces, logs, metrics, and audit records.

Operational signals should include:

• End-to-end and per-step latency.
• Input/output/cache tokens and estimated cost.
• Tool latency, errors, denied calls, retries, and duplicate prevention.
• Context size, truncation, retrieval hits, and citation failures.
• Loop depth, handoff count, queue time, cancellation, and budget exhaustion.
• Safety-policy triggers and approval wait time.
• Model/provider/version, prompt/spec/tool versions, and release cohort.
• Verified business outcome, not merely “agent completed.”

Redact or tokenize sensitive fields before export. Keep audit evidence protected from ordinary model context. Because telemetry conventions are evolving, use an internal semantic layer so vendor or OpenTelemetry field changes do not rewrite the whole application.

Evidence: S22, S77, S90

15.14 Reliability, performance, and cost

Agent loops multiply calls and dependencies. Bound the system explicitly:

• Maximum turns, wall-clock time, tokens, cost, delegations, and concurrent tools.
• Per-tool timeout, retry policy, idempotency, and circuit breaking.
• Total deadline propagated to child work.
• Backpressure and fair scheduling across users/tenants.
• Fallback or graceful degradation when model, retrieval, or tools fail.
• Checkpoints for long work and safe resume after interruption.
• Caching only where identity, freshness, privacy, and nondeterminism permit it.
• Cancellation that actually stops queued and delegated work.

Measure cost per successful outcome rather than cost per model call. A cheap model that causes more retries, tool calls, or human corrections may cost more overall.

Evidence: S17, S18, S23, S44, S64, S68, S69

15.15 Agent lifecycle

1. Define business outcome, users, scope, autonomy, and risk tier.
2. Build the smallest deterministic or workflow baseline.
3. Specify behavior, tools, context, memory, security, and evaluations.
4. Prototype in an isolated environment with synthetic/non-sensitive data.
5. Run repeated offline evaluations and inspect failure trajectories.
6. Threat-model and adversarially test the full data/action path.
7. Pilot with read-only or approval-gated actions and a narrow user cohort.
8. Canary model, prompt, tool, retrieval, and policy changes independently where possible.
9. Monitor verified outcomes, safety, reliability, latency, and cost.
10. Feed incidents and evaluation failures back into the specification and regression set.

Evidence: S24, S25, S68, S81, S83, S84