How to scope an AI red-team exercise without exposing real customer data

Published 8 November 2026

An AI red-team exercise should start with written permission, a defined test environment and clear boundaries on what testers and the system may access. A demonstration of model behaviour is useful only if the exercise itself stays within those boundaries.

The NIST Generative AI Profile, published in July 2024, includes adversarial testing and red teaming among approaches for identifying risks. It provides a risk-management reference; it does not grant permission to test someone else’s systems.

The checklist below is an operational proposal for a team planning a contained evaluation.

Write down the authorised scope

Name the person commissioning the exercise and the person who can stop it. Identify the model, application version, test accounts and tool connections. List the permitted systems and explicitly exclude production systems and third-party services unless their owners have authorised the relevant testing.

Define the question you are trying to answer. For example, can an assistant recognise that a document contains an instruction it should not follow? That question needs a narrower environment than an open-ended attempt to make the system act unpredictably.

Prepare data that can safely fail

Create synthetic customers, documents and credentials for the exercise. Mark them as test material and check that they do not correspond to real people or working accounts. Avoid importing a production database simply because it is convenient.

Give each test account only the permissions it needs. A tool that sends messages should point to an approved test destination, with external delivery disabled or restricted through an enforced control. A tool that changes records should operate on disposable test records. Review these boundaries with the owner of the connected application.

Agree stop conditions before the first test

• Unexpected contact with a production or third-party system.
• Exposure of real personal information or a working secret.
• An attempted action beyond the written scope.
• Unexpected cost, resource use or loss of control of the test environment.

If one occurs, stop the exercise, preserve the relevant evidence and notify the designated owner. Do not broaden the test to see how far the behaviour can go. The next step is a scoped investigation with the appropriate authority.

Record enough evidence to reproduce the finding

Keep the test objective, system version, relevant inputs, tool permissions, expected result and observed result. Store sensitive logs in the approved location and redact them before sharing a report. Distinguish the model’s text from actions a connected tool actually executed.

Our explanation of access-control responsibilities can help assign the owner of each connection. Teams evaluating AI in operational settings can also use the questions in our AI-governance guide for microfinance banks to discuss oversight before deployment.

When is the exercise complete?

Close a finding only after the responsible owner has reviewed the fix and the relevant test has been repeated. Record any remaining limitation. A passed test demonstrates the result for that test and environment; it is not a guarantee that the system cannot fail in another setting.

Featured image: AI-generated conceptual illustration.