What Does an AI Penetration Test Cover?

A customer-support assistant reads a ticket containing an instruction that was never meant for the user to see. The model treats that text as trusted guidance, calls an internal search tool, and returns information from the wrong customer account. Nothing “hacked the model” in a cinematic sense. The failure happened because untrusted content, model reasoning, tool permissions, and application authorization were connected without enough control.

That is the kind of problem an AI penetration test is designed to examine.

The assessment does not merely ask whether a chatbot can be persuaded to say something strange. It tests the AI-enabled workflow: prompts, retrieval sources, APIs, identity boundaries, tools, model outputs, logging, and safeguards around actions. It also defines what is outside scope, because an AI pentest is not automatically a model-quality audit, compliance certification, source-code review, or infrastructure assessment.

The guide explains what an AI penetration test covers, excludes, and delivers.

What Does an AI Penetration Test Cover? The Practical Answer

An AI penetration test covers the security of an AI-enabled application as a system, not only the language model.

That system can include:

• User prompts and uploaded files
• System instructions and prompt templates
• Retrieval-augmented generation, or RAG
• Vector databases and knowledge stores
• LLM APIs and orchestration frameworks
• Application APIs and authentication
• Agent tools, plugins, and external services
• Session state, memory, and conversation history
• Output rendering and downstream automation
• Logging, monitoring, rate limits, and cost controls

The exact scope depends on what the AI can access and what it can do. A public FAQ bot with no private data has a smaller attack surface than an internal copilot that searches customer records. An agent that can create tickets, send email, update a CRM, or execute cloud workflows requires deeper authorization and abuse testing than a read-only assistant.

Pentest Testing Corp scopes these components before testing begins through its AI penetration testing service. The goal is to produce a defensible test boundary rather than applying a generic prompt list to every system.

AI Pentest Scope Checklist

A useful AI pentest scope checklist should cover the layers below. Not every engagement needs every layer, but each one should be considered and explicitly included or excluded.

1. AI Architecture and Trust-Boundary Mapping

Testing starts with an architecture review focused on data flow and control flow.

The assessor identifies:

• Where prompts originate
• Which inputs are trusted, semi-trusted, or untrusted
• Which model or model gateway processes the request
• What context is added before the model call
• Which retrieval sources are queried
• What tools or functions the model can invoke
• Which user identity and permissions are applied
• Where outputs are stored, displayed, or executed
• Which third parties receive prompts, files, or metadata

This step is essential because the visible chat interface is often only the first component. The security-relevant behavior may happen in an orchestration service, a vector search query, an authorization middleware, or a tool connector.

A scope that says only “test the chatbot” is usually too vague. A stronger scope names the model endpoint, retrieval layer, connected tools, user roles, data classes, and permitted test environments.

2. Direct and Indirect Prompt Injection Testing

Prompt injection is mapped to OWASP LLM01:2025 Prompt Injection. OWASP defines this risk around inputs that alter model behavior or output in unintended ways. Retrieval and fine-tuning may improve relevance, but they do not remove the need to test instruction boundaries.

An AI penetration test normally examines:

• Direct user attempts to override intended behavior
• Indirect instructions embedded in documents, tickets, web pages, email, or retrieved content
• Conflicts between system, developer, tool, and user instructions
• Whether untrusted content is clearly separated from executable instructions
• Whether safeguards depend only on wording in the system prompt
• Whether the model can be induced to reveal restricted data or take unauthorized actions

The assessment should not be scored by how many “jailbreak phrases” produce unusual output. The important question is whether manipulated input creates a meaningful security impact.

For example, getting a model to discuss a prohibited topic may be a policy problem. Getting it to retrieve another tenant’s data, expose a secret, or trigger an unauthorized workflow is a security finding.

3. Retrieval-Augmented Generation and Knowledge-Base Security

RAG systems add a large and often misunderstood attack surface.

Testing may cover:

• Access control around documents and collections
• Tenant isolation in vector searches
• Metadata filters and authorization enforcement
• Retrieval of deleted, archived, or restricted content
• Poisoned or misleading content entering the knowledge base
• Indirect prompt injection inside indexed documents
• Sensitive data exposure through similarity search
• Insecure ingestion pipelines
• Leakage between development, staging, and production indexes
• Permissions on vector database APIs and storage

A model may be functioning exactly as designed while the retrieval layer returns information the current user should never receive. That is not a model failure. It is an application authorization failure expressed through an AI interface.

For multi-tenant SaaS platforms, testing should verify that tenant ownership is enforced before content is added to the prompt. Filtering only after generation is too late.

4. Agent Tools, Function Calling, and Excessive Agency

When an LLM can call tools, security testing must move beyond generated text.

This area maps directly to OWASP LLM06:2025 Excessive Agency, which addresses systems that can invoke functions, plugins, or connected services and may perform harmful actions when the model receives manipulated, ambiguous, or incorrect instructions.

An AI pentest may test:

• Whether the agent has more tools than it needs
• Whether tools expose overly broad functions
• Whether the model inherits excessive user or service-account privileges
• Whether read actions can be escalated into write actions
• Whether confirmation is required before high-impact operations
• Whether tool parameters are validated server-side
• Whether the system enforces deny-by-default authorization
• Whether one tool’s output can manipulate the next tool call
• Whether repeated autonomous actions can exceed intended limits
• Whether tool errors fail safely

The model should never be the final authorization authority. A tool must independently verify that the authenticated user is permitted to perform the requested action.

Human approval is useful for high-impact steps, but it should not be treated as the only safeguard. Approval dialogs must show accurate, specific information about the action being approved.

5. Authentication, Authorization, and Session Security

AI features often sit inside existing web and API platforms. That means standard security controls remain in scope.

Testing can include:

• Authentication flows
• Session and token handling
• Role-based access control
• Object-level authorization
• Cross-tenant data isolation
• API key storage and rotation
• Service-account permissions
• Conversation ownership
• Shared-link access
• File and knowledge-base permissions
• Logout and token revocation
• Rate limiting and abuse prevention

A traditional API issue such as broken object-level authorization can become more serious when an assistant makes inaccessible records discoverable through natural-language queries.

AI penetration testing should therefore overlap with web and API testing where the AI feature depends on those components. The engagement must clarify whether this overlap is limited to AI-related endpoints or includes the broader application.

6. Sensitive Information Exposure

The assessment should identify sensitive information that may enter or leave the AI workflow.

This can include:

• Customer records
• Personal data
• Authentication tokens
• API keys
• Internal prompts
• Proprietary documents
• Source code
• Financial information
• Health information
• Support conversations
• Model provider request and response logs

Testing examines whether sensitive data appears in prompts, responses, tool calls, logs, memory, analytics, or third-party model platforms.

It also verifies whether role and tenant restrictions are preserved when the model retrieves, summarizes, or transforms information. Masking a field in the user interface does not help if the raw value remains available to the model or tool layer.

7. System Prompt and Configuration Exposure

A system prompt can contain internal policy, routing rules, business logic, or operational detail. Exposure may be relevant, but it is not automatically critical.

An AI pentest should distinguish between:

• Disclosure of harmless instructions
• Exposure of proprietary business logic
• Leakage of credentials or secrets
• Instructions that reveal bypass conditions
• Internal endpoint names or tool schemas
• Hidden data sources or retention behavior

The severity depends on impact. A prompt that says “be concise and professional” is not equivalent to one containing a private API key or a rule that can be manipulated to bypass authorization.

The better remediation is not simply “hide the prompt.” Sensitive secrets should never be stored there, and security controls should remain effective even when prompt content is known.

8. Output Handling and Downstream Use

Model output is untrusted input to the next component.

Testing may assess whether generated content is:

• Rendered as HTML
• Inserted into templates
• Passed to a shell or interpreter
• Used in database queries
• Sent to external systems
• Parsed as structured JSON
• Used to select tool arguments
• Stored in logs or customer records
• Presented as a trusted recommendation

The application should validate, encode, constrain, and authorize model output according to where it is used.

An LLM response that remains plain text has a different risk profile from one that becomes an executable command, workflow definition, database filter, or administrative action.

9. Abuse Resistance, Rate Limits, and Resource Consumption

AI systems can create cost and availability risks even without a conventional exploit.

Testing can examine:

• Per-user and per-tenant usage limits
• Token and request caps
• File-size and document-count limits
• Recursive or repeated agent loops
• Expensive retrieval patterns
• Model fallback behavior
• Concurrency controls
• Denial-of-wallet exposure
• Alerting for unusual consumption
• Separation between free and paid usage tiers

The objective is controlled validation, not stress testing production without authorization.

Any load-related test must have agreed thresholds, stop conditions, and monitoring contacts. A safe pentest proves the control weakness without creating operational harm.

10. Logging, Monitoring, and Incident Evidence

Security controls are incomplete when the organization cannot see abuse.

An AI pentest can review whether logs capture:

• User and service identity
• Prompt and tool-call correlation IDs
• Retrieval sources
• Tool names and parameters
• Authorization decisions
• Model and policy versions
• Guardrail events
• High-risk approvals
• Rate-limit events
• Data-access failures
• Administrative changes

Logging should avoid creating a new data leak. Full prompts and responses may contain sensitive data, so retention, redaction, access control, and encryption must be considered.

The tester may also validate whether realistic abuse creates actionable alerts rather than only raw log entries.

Traditional Penetration Test vs AI Penetration Test

The two assessments are complementary. An AI pentest does not make web, API, cloud, or mobile testing obsolete. It adds coverage for the model-mediated parts of the system.

For a deeper comparison, see AI penetration testing vs traditional penetration testing.

Sanitized Illustrative Attack Scenario

The following scenario is illustrative only. It does not contain a working payload, real client data, or operational exploitation steps.

A document-review assistant is allowed to read uploaded files and create follow-up tasks in a project-management system. A user uploads a document containing hidden text that attempts to redefine the assistant’s objective. The assistant summarizes the visible content correctly but also prepares a task containing information from another document in the same workspace.

A well-scoped AI pentest would examine several independent controls:

1. Whether uploaded documents are treated as untrusted data rather than instructions.
2. Whether retrieval is limited to documents the current user may access.
3. Whether the assistant can create tasks without explicit confirmation.
4. Whether task creation validates the user’s project membership.
5. Whether the generated task is previewed before submission.
6. Whether logs show the document source, retrieval decision, and tool action.

The finding would not be “prompt injection exists” in isolation. The report would describe the security impact created by the combination of instruction confusion, weak document authorization, and excessive tool permissions.

What an AI Penetration Test Does Not Cover Automatically

A professional scope should state exclusions as clearly as inclusions.

It Does Not Guarantee the Model Will Never Hallucinate

Hallucination and factual reliability may be evaluated where they create a security consequence, but an AI pentest is not a complete model-accuracy or product-quality assessment.

It cannot prove that every future answer will be correct.

It Is Not a Full Responsible-AI Audit

Bias, fairness, explainability, accessibility, workforce impact, intellectual-property risk, and broader governance may require separate specialists and assessment methods.

Security findings can support these programs, but the pentest is not a substitute for them.

It Is Not a Compliance Certification

An AI penetration test may produce evidence relevant to SOC 2, ISO 27001, HIPAA, PCI DSS, or customer security reviews. It does not issue certification or guarantee compliance.

The NIST AI Risk Management Framework is voluntary, flexible, and designed to help organizations manage AI risk. A pentest can provide evidence for risk measurement and treatment, but it covers only part of the broader governance process.

It Does Not Include Every Web, API, Cloud, or Network Asset

The AI feature may depend on a large platform. Unless the statement of work says otherwise, testing may be limited to the AI-related endpoints and integrations.

A full-stack engagement should list the traditional application and infrastructure components separately.

It Does Not Include Source-Code Review Unless Stated

Black-box and gray-box testing can identify exploitable behavior without reviewing code.

A source-assisted review can improve coverage of authorization logic, prompt construction, tool definitions, and security controls, but it must be explicitly included.

It Does Not Prove the Absence of All Vulnerabilities

Penetration testing is time-bound and sample-based. Models are probabilistic, systems change, and new content can create new behavior.

The deliverable should explain the tested versions, roles, environments, tools, and limitations.

It Is Not Continuous Monitoring

A point-in-time assessment cannot replace operational logging, alerting, secure development, regression testing, and periodic retesting.

Material changes to models, prompts, tools, data sources, permissions, or orchestration can justify a new assessment.

What LLM Pentest Deliverables Should Include

Useful LLM pentest deliverables should help engineers fix problems and help leadership understand risk.

A complete package normally includes:

Scope and Architecture Summary

This identifies tested environments, model providers, endpoints, roles, retrieval sources, tools, exclusions, and test dates.

Executive Summary

The executive summary explains the most important attack paths, affected business processes, aggregate risk, and remediation priorities without relying on security jargon.

Findings Summary

Each finding should have a unique ID, title, severity, affected component, status, and framework mapping.

Detailed Technical Findings

Each finding should include:

• Clear description
• Security impact
• Affected component
• Sanitized evidence
• Safe reproduction guidance
• Preconditions
• Severity rationale
• Relevant CWE or framework mapping
• Specific remediation
• Validation criteria

Attack-Path Analysis

Single weaknesses often become serious only when combined.

The report should show how untrusted content, retrieval access, model behavior, tool permissions, and missing confirmation controls create an end-to-end path.

OWASP and NIST Mapping

Findings can be mapped to the OWASP Top 10 for LLM Applications 2025 and, where useful, to NIST AI RMF functions or organizational controls.

Framework mapping should support decision-making, not replace technical evidence.

Remediation Roadmap

The roadmap should group actions into immediate, short-term, and longer-term work.

High-value fixes often include server-side authorization, least-privilege tools, action confirmation, retrieval filtering, output validation, secrets management, and better logging.

Retest Results

A retest should verify that the original attack path is closed and that the fix did not simply block one phrase or example.

A robust fix should work across equivalent inputs, roles, content sources, and tool paths.

Pentest Testing Corp’s redacted reporting standard uses prioritized findings, evidence, impact, remediation guidance, framework mappings, and retest notes. Organizations can review the assessment approach through the AI penetration testing service page.

How to Scope an AI Security Test Correctly

Before requesting a quote, prepare the following information:

• What the AI feature does
• Whether it is public, internal, or customer-facing
• Model provider and model family
• Production and staging environments
• User roles and tenant structure
• APIs used by the AI layer
• RAG sources and vector stores
• File-upload and ingestion paths
• Tools and actions available to the model
• Service accounts and permission levels
• Sensitive data categories
• Existing guardrails and approval steps
• Logging and monitoring available during testing
• Recent material changes
• Compliance or customer deadlines
• Whether source code will be available
• Whether testing can use dedicated accounts and test data

The scope should also define safety controls:

• Authorized environments
• Allowed test hours
• Prohibited actions
• Data handling requirements
• Rate and cost thresholds
• Named emergency contacts
• Stop conditions
• Rules for destructive or irreversible functions
• Evidence retention and deletion requirements

These details reduce ambiguity and help the tester spend time on meaningful attack paths rather than discovering basic architecture during the testing window.

Architecture and integration depth also determine scheduling. The AI penetration test timeline guide explains how scope affects preparation, testing, reporting, and retesting.

For early-stage teams, scope should be driven by what the AI can access and do, not by employee count. Our guide to AI penetration testing for startups explains when a focused assessment becomes necessary, how to prioritize LLM01 and LLM02 risks, and how to test meaningful attack paths without purchasing more coverage than the product requires.

Conclusion

The answer to “what does an AI penetration test cover?” is broader than prompt injection.

A serious assessment examines the complete path from untrusted input to model behavior, retrieved data, tool use, application authorization, downstream output, and operational evidence. It also documents exclusions so stakeholders do not assume that model quality, compliance, source code, cloud infrastructure, and continuous monitoring were tested when they were not.

The strongest engagements are scoped around real capabilities and real business impact. They test whether the AI can access something it should not, reveal something it should protect, or perform an action the current user is not authorized to approve.

Book a 30-minute AI pentest scoping call to define the system boundaries, roles, retrieval sources, tools, and deliverables before testing begins.