ProcessCyber Defense

AI red teaming: scope, attack, triage and retest, mapped to OWASP and MITRE ATLAS

AI red teaming is authorized adversarial testing of models, agents and the systems around them. A repeatable process has seven steps: scope and authorize, build a threat model, assemble an attack library, execute, triage, retest and report. Mapping the library to the OWASP list of LLM application risks and MITRE ATLAS gives findings a shared vocabulary and makes results comparable from one cycle to the next.

Reviewed 7 min read

On this page
  1. What an AI red team tests, and what it does not
  2. The seven steps from scope to retest
  3. The red team cycle at a glance
  4. Mapping test goals to OWASP's list of LLM application risks
  5. The vocabulary that keeps findings comparable
  6. Rules of engagement: what to settle in writing
  7. Authorization is part of the method
  8. Turning results into a program
  9. Questions and answers
  10. Sources

What an AI red team tests, and what it does not

The OWASP GenAI Red Teaming Guide, published on 22 January 2025, frames testing across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis1. That framing is useful because most serious findings sit outside the model weights. They appear where a model meets tools, data and permissions.

Red teaming differs from a benchmark. A benchmark measures how a model behaves on a fixed set of prompts. A red team behaves like an adversary: it chains steps, adapts to defenses and tries to reach something that matters, such as customer data or an unapproved action.

It also differs from a penetration test of the surrounding application, though the two overlap and are often run together. This guide covers the AI-specific work. Pair it with the layers described in prompt injection defense so that fixes have somewhere to land.

The seven steps from scope to retest

Each step produces an artifact that the next one needs.

  1. Scope and authorize

    Name the systems, environments and data in scope, the people authorized to test them and the actions that are off limits. Get the system owner's written approval. Test only systems you own or are authorized to test, and use production data only with explicit agreement.

    Output
    Signed scope and rules of engagement
    Owner
    System owner and red team lead
  2. Build the threat model

    Describe what the system can read, what it can do and what an attacker would want. List entry points such as user prompts, retrieved documents, email, plug-ins and connected tools. Rank scenarios by the harm they could cause.

    Output
    Ranked attack scenarios per system
    Owner
    Security architect
  3. Assemble the attack library

    Select test cases for each scenario and map each to a recognized category, such as an OWASP LLM risk or an ATLAS technique. Include both single-turn prompts and multi-step chains against tools. Version the library so retests use the same cases.

    Output
    Versioned library with category labels
    Owner
    Red team lead
  4. Execute

    Run the cases, then improvise where the system pushes back. Log every prompt, response, tool call and timestamp. Stop and notify the owner if testing risks real harm, such as unexpected access to live customer records.

    Output
    Complete test log with evidence
    Owner
    Red team
  5. Triage

    Reproduce each finding, remove false positives and rate it on impact and ease. Group duplicates by root cause, such as an over-broad tool permission, so one fix closes several findings.

    Output
    Rated, deduplicated finding list
    Owner
    Red team lead and engineering lead
  6. Fix and retest

    Assign each finding an owner and date, then rerun the same cases after the change. A finding is closed only when the retest passes. Keep failed retests visible rather than reopening them quietly.

    Output
    Closed findings with retest evidence
    Owner
    Engineering owners
  7. Report and schedule the next cycle

    Give executives the exposure and the trend. Give engineers reproduction steps. Add new cases for attacks that were novel, and set the next cycle around material changes such as a new model, tool or data source.

    Output
    Two-level report and next-cycle plan
    Owner
    Red team lead

The red team cycle at a glance

01Scope andauthorize02Threat model03Attack library04Execute and log05Triage and rate06Fix and retest07Report and plan
  1. Scope and authorize

    Agree targets, rules of engagement and approvers.

  2. Threat model

    Rank attack scenarios by potential harm.

  3. Attack library

    Versioned test cases tagged to OWASP and ATLAS.

  4. Execute and log

    Run cases, adapt, and record every step.

  5. Triage and rate

    Reproduce, deduplicate and rate each finding.

  6. Fix and retest

    Close a finding only when the retest passes.

  7. Report and plan

    Brief leaders and engineers, then set the next cycle.

The cycle repeats after material changes to the model, tools or data.

Mapping test goals to OWASP's list of LLM application risks

OWASP 2025 riskWhat to attemptEvidence of a finding
LLM01 Prompt InjectionPlant instructions in documents, web pages and messages the system reads, and in direct prompts2.The system takes an action or reveals data the planted text asked for.
LLM02 Sensitive Information DisclosureAsk for data from other users, internal documents or earlier conversations2.Restricted content appears in a response or log.
LLM03 Supply ChainReview models, datasets, plug-ins and servers for unvetted or unsigned components2.A component loads without provenance or integrity checks.
LLM05 Improper Output HandlingMake the model emit markup, commands or queries and see whether downstream systems run them2.Output reaches a browser, shell or database unvalidated.
LLM06 Excessive AgencyTry to make an agent use tools or permissions beyond its task2.A high-impact action runs without approval.
LLM07 System Prompt LeakageTry to extract hidden instructions, keys or rules from the prompt2.Secrets or internal logic appear in output.
LLM10 Unbounded ConsumptionSend expensive or looping requests and measure limits2.Cost or availability degrades without a cap or alert.

OWASP's 2025 list also includes data and model poisoning (LLM04), vector and embedding weaknesses (LLM08) and misinformation (LLM09)2. Add them when your system trains on, retrieves from or relies on such data.

The vocabulary that keeps findings comparable

MITRE ATLAS
A knowledge base of adversary tactics, techniques, mitigations and case studies for AI systems, which tags entries by platform such as predictive, generative or agentic AI and by maturity such as feasible, demonstrated or realized3.
OWASP LLM application risks
A ranked list of the most important risks for applications built on large language models, updated for 2025 and used here to organize test goals2.
NIST AI 100-2 E2025
The March 2025 NIST taxonomy and terminology of adversarial machine learning attacks and mitigations, covering data poisoning, evasion, privacy and abuse4.
Rules of engagement
The written agreement that sets targets, permitted techniques, data handling, stop conditions and contacts before testing starts.

Rules of engagement: what to settle in writing

Settle these in writing during step one. They protect the testers, the business and the people whose data the system holds.

0 of 6 checked

Authorization is part of the method

Turning results into a program

A one-off exercise helps, but a cycle helps more. Rerun the versioned library when the model, tools, data sources or prompts change, and add new cases for each novel finding. Over several cycles, the count of open findings by OWASP category shows where design decisions keep producing the same weakness.

Reporting should serve two audiences. Leaders need exposure, trend and decisions required. Engineers need reproduction steps and the root cause. Both should trace back to the same finding IDs so that board discussions and fixes stay linked. For the leadership decisions around this work, see the AI security strategy page.

Questions and answers

What is AI red teaming?

It is authorized adversarial testing of AI systems, including models, agents and the data, tools and permissions around them. The OWASP GenAI Red Teaming Guide describes four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis1. The aim is to find weaknesses that matter before an attacker does.

How is AI red teaming different from a penetration test?

A penetration test targets conventional software and infrastructure weaknesses. AI red teaming adds model-specific attacks, such as prompt injection, data leakage through responses and manipulation of agents that call tools. In practice, the two are often scoped together because the serious findings tend to sit where the model meets the surrounding application.

Should we use OWASP or MITRE ATLAS to organize testing?

Use both for different jobs. OWASP's Top 10 for LLM Applications gives a ranked list of application risks that works well for planning test goals2. MITRE ATLAS catalogs adversary techniques and case studies that help describe how an attack unfolds3. Tag each test case with one label from each.

How often should we run an AI red team exercise?

Run one before a system first goes live and again after material changes, such as a new model, new tool access or a new data source. A recurring cycle keeps the versioned attack library retested. Frequency should follow the system's potential for harm rather than a fixed calendar.

Can automated tools replace human red teamers?

Automation is useful for breadth and regression testing of known cases. People are better at chaining steps, adapting to defenses and judging impact. A sound program uses tools to run the versioned library and humans to explore new paths and to triage and rate what the tools report.

Sources

  1. GenAI Red Teaming Guide — OWASP GenAI Security Project · checked 19 November 2026
  2. OWASP Top 10 for LLM Applications 2025 — OWASP GenAI Security Project · checked 19 November 2026
  3. ATLAS data: tactics, techniques, mitigations and case studies — MITRE ATLAS · checked 19 November 2026
  4. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025) — NIST · checked 19 November 2026

More in Cyber Defense

Back to Cyber Defense

Next step

Scope a first AI red team cycle

Share the AI system you would test first, the tools and data it touches and who owns it. We will propose a scope, rules of engagement and a draft attack library mapped to OWASP and ATLAS.

Scope a red team cycle

Your starting point

What would you like to move forward?

Start with the outcome that feels closest. We'll help give the next step a useful shape.

Let's find your next move

One conversation. A clearer direction.

Tell us what you want to improve. A few sentences are enough to start.

shayan@coldai.org