The company had deployed an AI fitness coach. Users could log their health goals, get meal plans, track workouts. The AI was scoped tightly: health and fitness topics only. No personal data crossing between accounts. No backend access. It was a clean product.
Angelina House and her team at Point Solutions Security tested it. They started by making the AI repeat the letter “A” 256 times, then 256 times that, then 256 times that again. The repetition overflowed the AI’s memory. The guardrails reset. Once they were gone, the team could ask it anything: dangerous dietary advice, instructions for accessing the backend language model, data from other clients’ accounts. No automated scanner would have found that path. It required a human who understood how that specific AI system processed instructions.
PSS provides manual pen testing services, so we have a perspective on this question. We’re going to give you the honest comparison anyway, including the situations where automated tools are the better call.
That’s not a knock on automated tools. It’s just not what they’re built for.
So what’s the difference between automated and manual pen testing? And which one is the right call for your business?
The Short Answer: Automated tools find known vulnerabilities fast and cheaply. Manual pen testing finds the chain that actually gets you breached. For compliance requirements, AI and LLM environments, and understanding your real attack surface, manual testing is not optional. Use both, not one instead of the other.
What Can Automated / AI Pen Testing Actually Do?
Automated scanning tools are fast. They can sweep a large environment in hours and produce a list of known CVEs (Common Vulnerabilities and Exposures), the catalogued weaknesses that already have names, IDs, and documented exploits. If you want to know whether your environment has unpatched software, known misconfigurations, or vulnerabilities that show up in public databases, automated tools are good at that.
PSS uses them internally for exactly this purpose. As Angelina puts it: we use AI to do first sweeps, pull out anything that stands out, and prioritize where to focus. Our pen test reports run to 80 pages. AttackForge, the platform PSS uses to document findings, generates the first draft of that report from what the team feeds it, which is AI doing what it’s built for. Paige Goss notes that a lot of the baseline OSINT work is also faster now with AI than it was before. Open source intelligence gathering in week one of an engagement used to take longer. Now it doesn’t.
So automated tools have a real place. They’re useful for first sweeps, for continuous CVE monitoring between manual engagements, and for making the manual team more efficient.
What they don’t do: follow the path beyond the entry point. They don’t pick up two unrelated vulnerabilities and ask whether chaining them together creates something worse. They don’t understand how your specific web application handles session tokens, or whether the way your AI chatbot processes repeated instructions will cause its safety rules to disappear.
Automated vs. Manual Pen Testing: At a Glance
| Automated / AI Scanning | Manual Pen Testing | |
| Speed | Fast — hours to days | Thorough — 2 to 4 weeks |
| Cost | Low to mid | Higher — reflects the depth of work |
| Known CVEs / catalogued vulnerabilities | Strong — this is what it’s built for | Covered — and confirmed by a human |
| Exploitation chains | Cannot do this | Core strength — this is the whole point |
| Business logic attacks | Cannot do this | Core strength |
| AI / LLM environment testing | Cannot do this | Yes — PSS does this |
| Evidence and remediation guidance | Partial — vulnerability list only | Full — screenshots, paths, implications, fix guidance, retest |
| Satisfies compliance auditors | Rarely | Usually required by framework [VERIFY WITH PSS] |
Why Can’t Automated Tools Replace a Human Tester?
The core difference is exploitation chains.
Chris Brown, PSS’s VP of Commercial Services, describes real pen testing this way: it’s a deep understanding of the infrastructure, attacking the weak points, seeing what you can exploit, and once something is exploited, seeing where you can go from there. An automated scanner identifies entry points. A human tester asks what you can reach from there, what connects to what, and whether two separate weaknesses add up to something that hands you the whole environment.
Paige Goss frames it directly: the human element adds a dynamic of putting multiple pieces of technology or multiple pieces of vulnerability together to create a unique attack chain. Technology supports the baseline, it sets the starting point. The human component adds the depth that reveals the most vulnerable areas and understands what an attacker would actually do once they’re inside.
The house analogy Chris uses is accurate. An external pen test is trying to break into your house. An internal pen test is: once you’re in, how many rooms can you get to and what can you take from those rooms? Automated tools check whether the front door is locked. A manual tester goes around back, checks the windows, tries the chimney. If they find a window cracked open, they go all the way in and document every room they reach.
Angelina’s team finds the same handful of issues on almost every engagement, and none of them show up in a CVE database. An unsecured printer connected to the main network. A few lines in an HTML script that allow login bypass without valid credentials. Generic passwords like admin1234 that survive because nobody thought to run a fuzzing script against the login page. These aren’t exotic attacks. They’re the small things that get overlooked because no scanner knows to look for them.
“We don’t do automated pen testing,” Angelina says. “Everything is manual. If we find something, we go down the rabbit hole.” .
Watch: We broke an AI fitness app’s brain. Here’s how.
Angelina walks through the real engagement on the fitness app AI. How the memory overflow worked, what the team could access once the guardrails came down, and what the fix actually looks like. If you have an AI tool deployed at your company, this clip is worth four minutes of your time.
When Are Automated Tools the Right Call?
Automated tools are genuinely useful, and there are situations where they’re the right choice.
If your company needs broad CVE coverage across a large environment and budget is a real constraint, continuous automated scanning between annual manual engagements is better than nothing. If you want to know whether a newly deployed system has obvious known vulnerabilities before your team touches it, a quick automated sweep is reasonable. If you’re monitoring for new CVEs against a stable environment, automated tools do that well and efficiently.
The honest position is that automated scanning and manual pen testing aren’t competing products. They serve different purposes. A company running continuous automated monitoring plus one or two manual engagements per year is better positioned than a company doing either one in isolation.
What they’re not interchangeable for: compliance frameworks that specify manual testing, any environment where business logic is part of the attack surface (web applications, mobile apps, AI tools), and any situation where the actual question is how far could an attacker get if they were in my environment right now. That question requires a human to answer it.
How PSS Approaches This and What AI Changes
PSS’s testing is 90% or more manual. That’s not a philosophical position against AI. It’s a reflection of what manual testing actually finds that automated tools don’t.
The fitness app test is the clearest example. A scanner gives you a list of ports and known CVEs. It does not make the AI type the letter A sixteen million times to see what happens when its memory fills up. That attack required human creativity about how that specific system’s guardrails worked. The fix: making guardrails survive a reset, building in keyword flags, adding clear deviation rules. This required understanding how the AI was processing inputs at a structural level. No scanner surfaces that.
Chris’s framing of AI’s broader role is useful here. AI levels up medium-tier adversaries, attackers who previously lacked the skill to execute sophisticated attacks can now automate their way to competence faster. On the defensive side, AI replaces Tier 1 triage work but does not replace the analysts who make judgment calls. He uses the Lenny analogy from Of Mice and Men: AI is strong and useful, but it needs someone with judgment directing it. Give a powerful tool to someone without the judgment to use it, and you get a different kind of problem.
The same logic applies to pen testing: AI tools amplify the human tester’s reach, but they don’t replace the human who decides where to look and what to chain.
The PSS 4-Week Engagement
Understanding what manual testing looks like in practice makes the comparison concrete. PSS runs most engagements over four weeks:
- Week 1 — OSINT: Open source intelligence gathering. What’s publicly findable about your environment before the team actively attacks anything. AI accelerates this phase considerably.
- Week 2 — Active Attack: Kick in doors. Try the windows. Go down the chimney if there’s one. No predetermined script, and the team follows what they find. This is where exploitation chains get built.
- Week 3 — Report and Readout: Here’s what we did, here’s the screenshot evidence, here’s the implication if this were a real attacker, here’s how to fix it so it can’t happen again.
- Week 4 — Retest: PSS retests after remediation. Did the fix actually work? This is included in the engagement, not an add-on.
Automated tools don’t do the report-and-retest loop with human judgment attached. They produce a list. The human work, which involves deciding what matters most, explaining what an attacker would actually do with that finding, helping the client understand the implications, that’s not automatable yet.
As Paige puts it: PSS uses AI “as a back end for us and helping expedite a lot of the mundane nature so that we can use our humans as the experts that they are, really diving in and putting chains of attack together versus just sort of pounding at the external basis of an organization.”
How Do I Decide What We Actually Need?
The honest framework, based on where most companies actually are:
| Your Situation | What to Do |
| Never had a manual pen test | Start there. Automated tools tell you what’s known. A manual test tells you what’s actually exploitable — and how far an attacker would get. |
| Compliance requirement (SOC 2, CMMC, PCI-DSS, HIPAA) | Manual testing is almost certainly required. Automated scans will not satisfy auditors for most frameworks. [VERIFY WITH PSS: confirm which specific frameworks require manual vs. accepting automated scans] |
| Want continuous coverage between annual engagements | Layer automated scanning on top of manual work — not instead of it. Use automated tools to catch new CVEs between engagements. Use manual testing to know what’s actually exploitable. |
| Have a web app, mobile app, or AI tool in production | Manual testing is not optional. Business logic and AI guardrail attacks require a human who can think laterally about how your specific system works. No scanner will test for what it doesn’t know to look for. |
| Budget is constrained this year | One manual test per year is the minimum. If budget is the constraint, one real engagement beats twelve months of automated scanning. Prioritize external and internal. Add web app testing once you have the baseline covered. |
On frequency, Chris’s guidance is direct: “Two a year is ideal. One a year is fine. Zero is not great.” Technology changes fast. A 12-month-old environment has usually shifted enough that the attack surface looks different. If you can only do one, do it. But build toward two.
Frequently Asked Questions
AI/automated tools scan for known vulnerabilities fast and cheaply. Manual testing uses human testers who chain vulnerabilities together to simulate a real attack. Automated tools find the door. Manual testers find out how far in you can get.
No. Automated tools identify known CVEs and provide useful continuous monitoring. They cannot build exploitation chains, test business logic, or think like an attacker. For compliance requirements and real-world risk assessment, manual testing is not optional.
Two per year is ideal. One is the minimum. Zero is not great. Technology changes fast enough that a 12-month-old environment has almost certainly shifted — and your attack surface with it.
Yes — for specific purposes. Automated tools are useful for continuous CVE monitoring and first-sweep triage between manual engagements. PSS uses AI internally to accelerate OSINT and generate initial report drafts. The problem is using them as a substitute for manual testing.
Four types: external (breaking in from outside), internal (once inside, how far can you get and what can you reach), web app (customer portals, applications), and mobile. PSS recommends starting with external and internal. All testing is 90%+ manual.
Unsecured printers on the main network. Login bypasses buried in HTML scripts. Generic passwords like admin1234 that survive because nobody thought to fuzz for them. AI guardrail resets that expose backend access and cross-client data. The small things that aren’t in any CVE database but are always the way in.
Want to know exactly what a real pen test looks like? And how much does it cost?
This post explains what’s the difference between a vulnerability scan and a real pen test, what each one actually costs, and how to make sure you are getting what you are paying for.
What Real Pen Testing Looks Like And How Much Does It Actually Cost?
Cybersecurity as a Service isn’t a software subscription. It’s a team of people who treat your security like it’s their own problem. If you’re ready to know what you’re business’ vulnerabilities are, contact us so we can have a straight conversation. No pitch deck. No pressure.
Let’s get your house in order. Fill this out and tell us what’s your biggest security headache right now? >>> Click here
We’ll look at where you are, tell you the truth about what we find, and work with you on what to fix first. If you want a free dark web scan to see what’s already out there with your name on it, we’ll run that too.
