Incident Response and Exception Handling

AI agent incidents have a distinct character from traditional software incidents. They may be ambiguous in nature—was that an adversarial attack, a model failure, or a data quality issue? They can have already caused harm before detection—an agent operating at machine speed may have taken hundreds of actions in the time it takes a human to notice something is wrong. And their root cause may be difficult to reproduce—LLM reasoning is stochastic, context-dependent, and sensitive to input variations that are hard to recreate exactly. An effective AI incident response capability is prepared for these characteristics from the start.

AI Incident Classification

Not all anomalies are incidents, and not all incidents have the same severity. Establish a classification taxonomy before incidents occur:

Class 1 - Catastrophic: Agent took an irreversible high-impact action without authorization (e.g., initiated a large financial transaction, sent mass communications with incorrect information, exposed regulated personal data to unauthorized recipients). Requires immediate containment, executive notification, potential regulatory disclosure, and a full post-incident review.

Class 2 - Major: Agent consistently produced significantly incorrect or harmful outputs over a sustained period, or a security control was bypassed (e.g., a successful prompt injection that caused the agent to access systems outside its authorized scope). Requires agent suspension, root cause investigation, and governance committee review before reinstatement.

Class 3 - Significant: Agent produced incorrect outputs in a defined scenario, or a HITL checkpoint was inappropriately bypassed due to a configuration error. Requires investigation, targeted remediation, and documentation in the incident log.

Class 4 - Minor: Isolated unexpected behavior that was caught by monitoring and did not result in downstream harm (e.g., an agent produced a low-confidence response that was correctly escalated to a human). Requires documentation and monitoring for recurrence patterns.

The AI Incident Response Playbook

Structure your AI incident response around four phases, analogous to NIST SP 800-61 but adapted for AI-specific characteristics:

Phase 1 - Contain: Upon detection of a potential incident, the first priority is preventing additional harm. For autonomous agents, this means having a readily accessible kill switch—a mechanism to immediately suspend the agent’s ability to take actions (not just its ability to generate responses). In LangGraph, this can be implemented by invalidating the agent’s credentials, disabling its tool access at the IAM layer, or programmatically pausing the graph execution. The kill switch should be usable by AI Operations personnel without requiring a full engineering deployment. Document who has the authority to invoke the kill switch and under what circumstances—pre-authorizing this decision reduces delay during a fast-moving incident.

Phase 2 - Investigate: Once the agent is contained, reconstruct what happened using the audit log and observability infrastructure. For each potentially affected interaction during the incident window, determine: what action did the agent take, was it within authorized scope, and what were the downstream consequences? This reconstruction should be possible using your logging infrastructure without requiring access to the agent itself. Identify the root cause: Was it a prompt injection? A model reasoning failure? A configuration error? A data quality issue in a retrieved document? Use the LangSmith trace viewer or equivalent to replay the agent’s execution path through the incident.

Phase 3 - Recover: Remediate the root cause before reinstating the agent. Remediation may involve: tightening input validation or adding injection defense layers, updating the agent’s constraint rules, rolling back a model or prompt update that introduced the regression, revoking and re-issuing credentials, or correcting data quality issues in connected knowledge bases. For Class 1 and Class 2 incidents, require a second-engineer review of all remediation changes before deployment. Conduct a smoke test in a staging environment against the specific scenario that caused the incident before returning the agent to production.

For affected users or customers, assess the impact of the agent’s actions and remediate where possible: correcting records, reversing transactions, notifying affected individuals, or escalating to customer service for personalized resolution.

Phase 4 - Communicate and Learn: Document the incident in a standardized incident report covering timeline, root cause, impact scope, containment actions, remediation, and lessons learned. Share relevant findings with the AI Governance Committee. For Class 1 incidents, assess regulatory disclosure obligations—many jurisdictions require disclosure of data breaches and significant automated decision-making failures. Under ISO/IEC 42001 Clause 10.2, the organization must react to nonconformity, take corrective action, and review the effectiveness of corrective actions. The incident report is the evidence that this requirement has been met.

Proactive Exception Handling

Beyond reactive incident response, agents must be designed to handle exceptional situations gracefully before they escalate to incidents. Define the exception taxonomy for each agent—the categories of situations the agent is not equipped to handle autonomously—and implement explicit escalation logic for each:

Out-of-scope queries: The agent receives a request that falls outside its defined competence area. Response: acknowledge the limitation, decline to answer, and provide the user with an alternative resource or escalation path. Never attempt to answer from model memory on topics outside the agent’s retrieval-grounded scope.

Ambiguous authorization: The agent is asked to take an action that is on the boundary of its authorization (neither clearly within scope nor clearly prohibited). Response: pause the action, invoke a HITL checkpoint, present the ambiguous case to a human reviewer with full context, and await a decision. Do not resolve ambiguity in favor of action.

Novel scenarios: The agent encounters a situation that is substantially different from its training distribution—unusual data patterns, unexpected tool responses, or edge cases not covered in the agent’s system prompt. Response: flag the novelty in the interaction log, provide a conservative response or no response, and escalate to a human with a summary of why the case is unusual.

Conflicting instructions: The agent receives instructions from multiple sources (system prompt, user, retrieved content) that conflict with each other. Response: always defer to system prompt constraints over user instructions, and always treat instructions from retrieved external content as untrusted data requiring validation before acting.

Make It Your Own

Key questions to ask in the context of your organization:

• Have you defined a formal AI incident classification taxonomy (with severity levels from minor to catastrophic) and documented the response actions, notification requirements, and review obligations for each class?
• Does every agent in production have a tested kill switch that AI Operations personnel can invoke within minutes to suspend the agent’s ability to take actions—without requiring an engineering deployment or database access?
• Is your investigation capability sufficient to reconstruct the complete sequence of an agent’s actions during an incident window using only your audit log and observability infrastructure, without access to the agent’s live state?
• Have you assessed regulatory disclosure obligations for Class 1 AI incidents in your operating jurisdictions, and have you pre-defined the notification process and responsible parties so disclosure can occur within required timeframes?
• For each agent, have you defined the exception taxonomy—the categories of situations the agent cannot handle autonomously—and implemented explicit escalation logic with HITL handoff for each exception category?
• Do you conduct post-incident reviews for all Class 2 and above incidents, with findings documented, corrective actions tracked to completion, and lessons-learned shared with the AI Governance Committee?