ChecklistRisk & Resilience
AI incident response plan: a checklist for failures that need no attacker
An AI incident response plan covers what your security playbook does not: a model giving wrong or harmful answers at scale, data leaking through retrieval, an agent acting beyond its permissions, or a provider update that silently changes behavior. It defines what counts as an AI incident, how to rate severity, which signals detect it, how to contain it without an attacker to evict, what evidence to keep and which notification duties to check.
On this page
- Why AI failures break conventional incident playbooks
- Six AI incident types and the first question each raises
- Raising AI incident severity: four escalation questions
- Detection signals your AI incident plan should name
- Containment levers to build before go-live
- Evidence to preserve for AI incident investigation
- Notification duties to check with counsel
- Hypothetical walk-through: an assistant shows another customer's order
- Decision rights and the post-incident review
- Questions and answers
- Sources
Why AI failures break conventional incident playbooks
Security incident processes assume an adversary: someone got in, so you find them, evict them and close the gap. Many AI incidents have no adversary. A customer assistant that starts quoting a wrong refund policy, or an agent that applies a discount to every order, is behaving as built, just badly. The usual triggers, such as an intrusion alert, may never fire.
AI failures also spread differently. One flawed prompt or model version affects every conversation that uses it, so the harm scales with traffic rather than with an attacker's effort. The plan therefore needs AI-specific definitions, detection and containment, while reusing the organization's existing incident command, communications and legal processes. NIST's incident response guidance frames response as part of overall cybersecurity risk management rather than a standalone activity, which is the right home for AI incidents too1.
Six AI incident types and the first question each raises
| Incident type | Typical trigger | First containment question | Notification angle to check |
|---|---|---|---|
| Harmful or inaccurate output at scale | Prompt, model or retrieval change | Can we pause or roll back the version? | Consumer protection, sector conduct rules |
| Data exposure through a model or retrieval | Broken access filtering in search or memory | Can we disable the index or feature? | Personal data breach rules |
| Unauthorized agent action | Tool permissions wider than intended | Can we revoke tools and reverse actions? | Contract, financial and conduct rules |
| Discriminatory outcome | Biased data or proxy features in a decision | Can affected decisions be identified and reviewed? | Equality, employment or credit rules |
| Provider-change regression | Upstream model update or deprecation | Can we pin or switch the model? | Customer contracts and service levels |
| Outage or severe degradation | Provider outage, quota or cost limits | Can the process run manually? | Service levels; resilience reporting where relevant |
Raising AI incident severity: four escalation questions
Start every AI incident at a default level and move it up when any answer is yes.
- If
People outside the organization were affected, or could be, beyond a handful of cases.
ThenRaise severity and involve communications and customer operations early.
Scale is what separates a defect from an incident with external consequences.
- If
The harm cannot easily be reversed, such as disclosed data, executed payments or decisions already communicated.
ThenRaise severity and bring in legal immediately.
Irreversible harm often carries notification duties with short deadlines.
- If
The incident may engage a regulator, such as a personal data breach, a significant service disruption or a serious incident involving a high-risk AI system.
ThenStart the regulatory assessment in parallel with containment, not after it.
Several reporting clocks run from awareness, not from root cause.
- If
The behavior is still spreading through traffic, connected systems or automated actions.
ThenContain first, even at the cost of service, then investigate.
Every minute of a scaled failure adds affected users.
Detection signals your AI incident plan should name
Containment levers to build before go-live
Evidence to preserve for AI incident investigation
Notification duties to check with counsel
General Data Protection Regulation (EU) 2016/679
EU and EEAApplies whenThe incident is a personal data breach2.
- Article 33: notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware, unless the breach is unlikely to result in a risk to individuals2.
- Article 34: inform the people affected without undue delay when the breach is likely to result in a high risk to them2.
Regulation (EU) 2024/1689 (Artificial Intelligence Act), with the Digital Omnibus on AI amendments
EUApplies whenA high-risk AI system is involved in a serious incident, once the high-risk obligations apply to it; the Digital Omnibus on AI moved those dates5.
Hypothetical walk-through: an assistant shows another customer's order
Decision rights and the post-incident review
Name four roles in advance. The incident commander, from the existing incident process, runs the response. The AI owner knows the system and can pull containment levers. Legal and privacy decide on notifications. Communications owns messages to customers and staff. The plan should say who may switch off a revenue-generating AI feature without waiting for approval; if nobody can, containment will be late.
The review asks what failed, why monitoring did not catch it sooner and which control would have prevented it. Its most valuable output is a set of new evaluation cases that reproduce the failure, added to pre-release and live testing. Near misses deserve the same review. To rehearse the decisions in this plan before a real event, run it through a cyber tabletop exercise.
Questions and answers
Who should own AI incidents in an organization?
Run them through the existing incident management process, with its incident commander, rather than creating a parallel structure. Within that process, each AI system needs a named AI owner who understands its behavior and controls its containment levers. Risk or AI governance teams set the taxonomy and review trends across incidents, while legal decides on notifications.
Do near misses with AI systems count as incidents?
Record and review them, even if they never reach a severity that triggers full response. A near miss, such as an agent proposing an action outside policy that a human reviewer rejected, shows a control working and a weakness upstream. Reviewing near misses is the cheapest way to find failures before customers do, and it supplies realistic evaluation cases.
Is a single hallucinated answer an AI incident?
Usually it is a defect to log and fix, not an incident. It becomes an incident when it affects many users, causes harm that is hard to reverse, such as a wrong instruction acted on in finance or health, or reveals a systemic cause like a broken retrieval source. Your severity questions should make that distinction explicit, so staff know when to escalate.
How is this plan different from an AI agent security checklist?
A security checklist covers preventive design controls before launch, such as least-privilege tools, input handling and approval steps. The incident response plan covers what happens when something goes wrong anyway: detection, containment, evidence, notification and learning. The two connect, because each incident review should feed new controls and tests back into the design checklist.
Sources
- NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management, a CSF 2.0 Community Profile — National Institute of Standards and Technology · checked 10 October 2026
- Regulation (EU) 2016/679 (General Data Protection Regulation) — EUR-Lex · checked 10 October 2026
- Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) — EUR-Lex · checked 10 October 2026
- AI Act Article 73: reporting of serious incidents — European Commission AI Act Service Desk · checked 10 October 2026
- Regulation (EU) 2026/1744 amending Regulation (EU) 2024/1689 (Digital Omnibus on AI) — EUR-Lex · checked 10 October 2026