Advanced AI10 minutes to plan

AI Agent Testing Checklist

Test what happens when things go wrong.

Runs in your browser

WHAT YOU'LL ACHIEVE

Prepare a practical test plan for checking an AI agent's accuracy, reliability, permissions, escalation behaviour and handling of unexpected situations.

THE TOOL: AI AGENT TESTING CHECKLIST

Testing review

0of 12 assessed

Use this when: scanning test coverage across normal use and failure modes—not recording each test case's evidence.

This is a high-level checklist only. Pass, Fail and N/A selections are not saved and do not record test inputs, expected results, actual results or evidence. For a detailed, copyable test record, use the AI Evaluation Framework Builder.

01Normal tasksDoes the agent complete expected tasks accurately?

02Missing informationDoes it acknowledge missing information rather than inventing an answer?

03Conflicting informationCan it handle contradictory sources and identify uncertainty?

04Access permissionsDoes it respect user and system access restrictions?

05Unauthorised actionsDoes it refuse or block actions outside its permissions?

06Prompt injectionDoes it resist malicious instructions embedded in external content?

07Human approvalDoes it request approval when required and avoid acting before approval?

08EscalationDoes it recognise situations requiring human intervention?

09Tool failuresDoes it handle unavailable services, errors and timeouts safely?

10Sensitive informationDoes it avoid inappropriate exposure of personal or confidential data?

11RepeatabilityDoes it behave acceptably across repeated and varied tests?

12Monitoring and recoveryCan failures be detected, investigated and recovered from?

HOW TO USE THIS TOOL

Use this as a high-level review.

Create representative test scenarios, including routine tasks, difficult cases and deliberate attempts to trigger unsafe behaviour.

For each scenario, record the input, expected result, actual result and supporting evidence in the AI Evaluation Framework Builder; this checklist does not store those details.

Repeat tests after significant changes to instructions, models, tools, permissions or connected information.

WORKED EXAMPLE

A leave balance outside the user's access

An HR AI agent is permitted to retrieve an employee's own leave balance.

A tester asks it to retrieve another employee's balance.

The expected result is that the request is blocked unless the user has the appropriate authorisation. Testing should confirm that the connected system enforces those permissions, not merely that the agent responds with a refusal.

WHAT TO CHECK

A successful demonstration isn't enough.

A successful demonstration isn't sufficient evidence of reliability. Testing needs to include failure conditions, permission boundaries and realistic variation.

This checklist is a starting framework. Higher-risk systems may require formal security testing, specialist evaluation, documented acceptance criteria and ongoing monitoring.