AI Pentesting Checklist for Web Apps: 15 Questions to Ask Before Choosing a Vendor
Don’t choose an AI pentesting vendor based on a demo.
Most agents look good on a vendor’s test app. The differences show up on yours: login flows they can’t navigate, vulnerabilities they can’t validate, coverage they can’t prove, and production risks they can’t control.
The free checklist gives you 15 checks to expose these gaps, the red flags to watch for, and a weighted scorecard to compare vendors on your own web app.
What’s Inside the Checklist?
15 checks to separate proven capabilities from vendor claims.
✓ See what the agent knows before it attacks
Find out whether it starts from a baseline of your live site or relearns it on every run, and how fast it reaches a first test.
✓ Test authenticated workflows
Find out whether an agent can clear MFA and multi-step logins without you turning MFA off, and test access across user roles.
✓ Demand proof of exploitability
Tell validated, reproduced findings apart from raw AI output and false positives. Check whether the evidence supports a PCI DSS 11.4 assessment.
✓ Expose coverage gaps
Ask vendors to show what they tested, what they skipped, and how they measure coverage.
✓ Evaluate production safety
Check whether guardrails run outside the AI model, what data the agent keeps or trains on, and who can stop a run.
✓ Verify that fixes hold
Find out whether the agent reruns the exact attack after a fix, not a generic rescan.
✓ Compare vendors with a structured scorecard
Score up to three vendors across 15 checks, with weighted criteria, recorded evidence, and pass/fail gates for data handling and guardrails.
[Excerpts from the full checklist]
