Guides5 min read
AI security audit or penetration test?
Buy a penetration test when a contract, insurer or regulator asks for one. Run an AI security audit in between: it reads all your code and can be repeated.
Mercury Labs
Buy a penetration test when a contract, insurer or regulator asks for one. Run an AI security audit in between: it reads all your code and can be repeated.
Most companies that run their own software need both, at different times. Buy a penetration test when a contract, an insurer or a regulator asks for one. Run an AI security audit in between, and before a release, because it reads all of your code and costs little enough to repeat.
A penetration test is certified people trying to break into your system by hand, for an agreed number of days, and writing a formal report of what they managed. An AI security audit is AI models reading your code and probing the parts of your system that face the internet, with an engineer confirming each finding before you see it.
It gives you skilled people thinking like an attacker. A good tester chains small weaknesses together, notices the odd behaviour a tool would pass over, and tries the things your own team assumed nobody would. At the end you hold a report that other organisations recognise.
That recognition is often the reason to buy one. In the UK, CREST accredits the companies that do this work, and the National Cyber Security Centre runs the CHECK scheme for testing government systems. If you take card payments, the card industry's standard expects regular testing. If a customer's contract or your cyber insurer asks for a penetration test, this is what they mean, and nothing else will satisfy them.
Its limit is time. Testers have a fixed number of days, so they sample your system rather than read all of it. And the report describes one moment. Software that changes every week is a different system a month later.
It gives you coverage and speed. A model can read every file in a codebase, and every library it depends on, in the time a person reads a handful. That matters because most breaches do not come from clever attacks. They come from ordinary mistakes: a page that forgets to check who is asking, a password left in the code, input that is trusted when it should not be, a library with a flaw that was fixed two versions ago.
Because it is quick, you can run it again whenever the code has changed a lot: before a release, after a burst of new features, or before an investor's due diligence. What you receive is a short list, most serious first, with the evidence for each finding and a recommended fix.
It has limits too, and an honest supplier says so. AI models raise false alarms, which is why a person has to confirm each finding. The result carries no certificate. It looks at your software, not at whether your staff would open a convincing email or whether the office door is locked. And no scan, by a person or a machine, finds everything.
AI models have become good at finding security flaws. In 2024 a Google research system found a real, exploitable flaw in SQLite, software that runs on billions of devices, before it was released. In 2025 the teams in the United States' AI Cyber Challenge built systems that found and fixed flaws across large open-source projects without a person in the loop.
The same tools are available to attackers. A weakness that once needed a specialist to find can now be found by anyone willing to point a model at your public website. That is the case for looking first.
| Option | When to use | Trade-offs |
|---|---|---|
| Penetration test | A contract, an insurer, a regulator or a card-payment standard asks for one, or you are about to launch something sensitive | Days of certified people's time. A snapshot of one moment. Samples the system rather than reading all of it |
| AI security audit | Between formal tests, before a release, after a lot of new code (especially code written with AI tools), or before due diligence | No certificate. Needs an engineer to remove the false alarms. Does not test people or process |
| Automated scanner | All the time, in the background, to catch known problems as they appear | Finds only what it has a rule for, and produces long lists someone has to sort |
| Cyber Essentials | You sell to the public sector, or want a baseline your customers recognise | Checks your organisation's basic controls, not the software you have written |
05 stages
What exactly may be tested? Agree it in writing.
Watchpoint
Testing a system without its owner's permission is an offence under the Computer Misuse Act, whatever the intention.
Who checks the findings before we see them?
Watchpoint
A raw list from any tool, AI or otherwise, is mostly false alarms.
Where does our code go?
Watchpoint
Sending source code to a public AI service may break your own contracts with customers.
What do we get for each finding?
Watchpoint
A finding without evidence and a fix is a worry, not a result.
Who will fix what is found, and by when?
Watchpoint
A report nobody acts on changes nothing.
If a contract or a regulator requires a penetration test, commission one from an accredited firm, and run an AI audit a few weeks before it. The audit clears out the ordinary mistakes, so the testers spend their days on the hard problems and not on the obvious ones.
If nobody requires one yet and you have software of your own, start with the audit. Fix the serious findings, run it again after your next big release, and decide on a formal test when a customer asks or the system starts holding data that would hurt to lose.