5 Questions to Ask Before Hiring an AI Penetration Testing Firm

A SaaS company we talked with last quarter had already checked the box. They'd paid a vendor for "AI penetration testing" on their customer-facing chatbot, gotten a clean report, and moved on. Then a prospect's security team asked a pointed question during due diligence: did the test cover indirect prompt injection through uploaded documents? The answer was no. The report the vendor delivered was a standard web application scan with the word "AI" added to the cover page. The API endpoints got tested. The model's actual behavior never did.

This happens more often than it should, because there's no license required to call yourself an AI red team firm, and the market is still sorting out who's doing real adversarial testing against the OWASP LLM Top 10 versus who's repackaging a checklist. If you're trying to figure out how to choose an AI penetration testing company for your own chatbot, copilot, or agent, the questions you ask during the sales call matter more than the pitch deck. Here are five that separate a firm that understands LLM security testing from one that's using the label to upsell a service they already had on the shelf.

Why "AI Penetration Testing" Doesn't Mean One Thing Yet

Traditional penetration testing has decades of shared vocabulary behind it. Everyone in the industry roughly agrees on what a web app pentest or a network pentest includes, because frameworks like OWASP's Web Top 10 and PTES have been stable for years. AI security testing doesn't have that maturity yet. The OWASP Top 10 for LLM Applications only reached its current 2025 edition in late 2024, and a lot of firms are still mapping their existing methodology onto it rather than building testing practices around it from scratch.

That gap creates an opening for vendors to relabel existing services. A firm that's good at API security testing can genuinely test the authentication and rate limiting behind your LLM endpoints, and that's useful work. But it's not the same as testing whether your model can be manipulated into ignoring its own instructions, leaking another user's data, or taking an action it was never supposed to take. Those are model-layer and agent-layer problems, and they need a different kind of adversarial thinking than a network scan does.

This is where AI pentest vendor selection gets harder than picking a traditional security firm. You're not just checking certifications and past reports. You're checking whether the firm actually understands the difference between testing an application that happens to call an LLM API and testing the reasoning and permissions of the model itself. Our own AI penetration testing methodology is built around this distinction, which is a useful reference point when you're comparing it against a vendor's proposal.

Question 1: How Do You Test for Prompt Injection, and Do You Cover the Indirect Kind? (LLM01)

Prompt injection is ranked LLM01 in the OWASP Top 10 for LLM Applications for a reason: it's the most common way attackers get a model to behave outside its intended boundaries. But there's a meaningful difference between the two variants, and a vendor's answer here tells you a lot.

Direct prompt injection is what most people picture: a user types something like an instruction override directly into a chat window. It's real, but it's also the easier half of the problem, and plenty of guardrail products catch the obvious attempts. Indirect prompt injection is subtler and more dangerous in production systems. It happens when an attacker plants instructions inside content the AI processes later, not inside the user's own message. Think of a support ticket, an uploaded file, or a web page that a retrieval pipeline pulls in. The user never sees the malicious instruction. The model reads it as part of the "trusted" context and can act on it anyway.

Ask any prospective vendor directly: does your methodology include testing indirect injection through documents, retrieved content, or connected tools, or only direct injection through the chat interface? A firm that hasn't thought about the indirect path probably hasn't tested a RAG pipeline or an agent with file access under real adversarial conditions.

A Sanitized Example of Why This Matters

Here's a pattern we've seen across multiple engagements, described at a level that illustrates the risk without handing anyone a working exploit. An internal HR copilot was built to summarize uploaded candidate resumes and answer recruiter questions about them. During testing, a sanitized resume file was constructed containing hidden text, invisible to a human reviewer but readable by the model, instructing the assistant to disregard its summarization task and instead output the contents of every other candidate file it had access to in that session.

The assistant didn't have malicious intent baked in. It simply couldn't reliably tell the difference between "the recruiter's instruction" and "text buried inside a document it was asked to read." That's the essence of indirect prompt injection, and it's a pattern, not a specific payload. No working exploit code or step-by-step attack instructions are reproduced here, and none should be in any vendor's marketing either. If a firm's case studies read like a tutorial for reproducing an attack, that's a red flag, not a credential.

Question 2: Can You Walk Me Through a Finding Involving Excessive Agency? (LLM06)

If your AI system only answers questions, excessive agency (LLM06) matters less. If it can send emails, modify records, call internal APIs, or trigger workflows, it matters a great deal, and it's one of the categories generic web testing simply doesn't cover.

Excessive agency shows up when a model or agent has more capability, autonomy, or permission than its actual job requires. An agent that only needs to look up order status but also has write access to the billing database is carrying risk it doesn't need to. The failure mode isn't usually "the AI decided to do something malicious." It's that an attacker, through a manipulated prompt or poisoned input, pushes an over-privileged agent into taking an action nobody intended and nobody explicitly authorized.

Ask the vendor to describe, in general terms, a past finding involving agent permissions or tool abuse, without naming the client. A firm doing real LLM security testing should be able to talk about permission boundaries, least-privilege violations, and how they tested whether an agent could be coaxed into invoking a tool outside its intended scope. If the answer stays vague or pivots back to "we test for prompt injection," that's usually a sign the firm hasn't done much agentic testing yet, since agent hijacking and prompt injection are related but distinct problems.

Question 3: Is the Engagement Fixed-Price and Scoped, or Open-Ended by the Hour?

This question isn't about security depth directly, but it tells you a lot about how mature the vendor's process is. AI systems vary enormously in complexity. A single chatbot wired to a third-party model API is a very different engagement from a multi-agent platform with proprietary fine-tuned models and a dozen connected tools. A firm that's actually scoped a lot of these engagements should be able to give you a fixed number once they understand your architecture, not an open-ended hourly estimate that grows as testing proceeds.

Open-ended hourly billing isn't automatically a scam, but it does shift risk onto you, the buyer, in a market where most people can't yet judge whether the hours billed reflect real depth of testing. A scoped, fixed-price proposal forces the vendor to actually understand your system before quoting, which is itself a useful filter.

The Traditional Pentest vs. AI Pentest Comparison

Part of the confusion buyers run into is assuming a recent web or API pentest already covers their AI systems. It usually doesn't, and the table below shows where the two disciplines diverge.

If a vendor's proposal doesn't clearly separate these two disciplines, or worse, presents a web app scan as sufficient AI coverage, ask them directly which line items in the table above their methodology actually addresses.

Question 4: Will You Prove Every Finding, or Just List a Severity Score?

A report that says "the model is vulnerable to jailbreaking, high severity" isn't useful on its own. It doesn't tell your engineering team what specifically happened, what the model was tricked into doing, or how to reproduce and fix it. Ask whether the firm provides proof-of-concept evidence for each finding: the sanitized interaction that demonstrates the issue, the business impact it maps to, and clear remediation guidance your team can act on without guessing.

This matters more in AI security testing than in traditional pentesting, because the space is newer and there's more temptation to pad a report with theoretical risks that were never actually demonstrated against your system. A firm confident in its findings will show you the proof. One that isn't will lean on generic severity language instead.

Question 5: How Do Your Findings Map to the Compliance Framework We're Being Audited Against?

If SOC 2, ISO 27001, or a customer's vendor security questionnaire is driving this project, ask the vendor directly how their AI penetration testing findings tie back to those controls. SOC 2's logical access and change management criteria increasingly get interpreted to include AI systems that touch customer data or make decisions on a company's behalf. NIST's AI Risk Management Framework provides the governance structure many auditors reference, and the OWASP Top 10 for LLM Applications is becoming the testing standard cited alongside it.

A vendor who can explain, specifically, how an LLM06 finding relates to a SOC 2 access control, or how their reporting format lines up with the NIST AI RMF's Govern and Map functions, is speaking the same language your auditor will. One who can only describe findings in isolated technical terms will leave you translating the report yourself before it's useful for compliance purposes.

Red Flags Worth Watching For

A few patterns show up repeatedly among firms that are newer to this space or stretching an existing service to fit the AI label:

The proposal never names a specific OWASP LLM Top 10 category by ID. Generic language like "we test AI systems for vulnerabilities" without referencing LLM01 through LLM10 usually means the methodology wasn't actually built around the framework.

The sample report reads almost identically to a web app pentest report with an AI section bolted on. Real AI-specific findings read differently, because the attack classes are different.

Pricing is either suspiciously flat regardless of system complexity, or entirely open-ended with no attempt to scope first. Both suggest the vendor hasn't developed a real process for sizing these engagements.

There's no mention of agentic systems, tool permissions, or RAG-specific risks at all, which suggests the firm's experience is limited to simple chatbot testing.

How to Choose an AI Penetration Testing Company: Put the Five Questions to Work

Choosing the right AI red team firm comes down to whether they can answer these five questions concretely, with specifics, rather than with marketing language borrowed from a general pentest pitch. A firm that can walk you through real prompt injection and excessive agency findings, quote you a fixed and scoped price, prove every finding with evidence, and connect the technical results to your compliance obligations is doing the actual work this discipline requires. One that can't is probably still figuring it out on someone else's engagement, possibly yours.

If you've already had a standard web or API pentest this year, it's worth checking directly whether it covered AI-specific risk at all. Most don't, by design, since they weren't scoped to. Our guide on AI penetration testing versus a traditional penetration test breaks down exactly what typically gets missed when AI systems are tested with a web app methodology instead of an AI-specific one.

Ready to Vet a Vendor, or Skip Straight to Testing?

If you're evaluating AI penetration testing firms right now, use these five questions as your filter before you sign anything. And if you'd rather skip the vetting process and talk directly to the team that will run the assessment, book a free 30-minute scoping call. We'll walk through your AI systems, tell you honestly whether adversarial testing makes sense yet, and if it does, scope a fixed-price engagement mapped to the OWASP LLM Top 10 from the first conversation, not after you've already paid for a report that missed the point.