What is penetration testing, anyway?

If you have a software system that protects valuable data or other assets, you probably want to have it tested for security vulnerabilities. That has probably led you to explore types of security assessments, and you’ve probably found that the most commonly referenced one is “penetration testing.”

What many companies don’t realize, however, is that “penetration testing” oftentimes isn’t really penetration testing at all. Worse yet, they don't realize that they actually might need something else.

When it comes to testing applications for security vulnerabilities, terms are used incorrectly all the time. If you don’t realize it’s happening, it can have dire consequences.

Download Ted's free ebook, "How to secure your software faster and better."

Get Your Copy

Most people ask for penetration testing, but are sold vulnerability scanning instead. However, what most people actually need is something else entirely: vulnerability assessments.

Those are remarkably different things. Each requires a different investment of time, effort and money. Each has different goals. Each produces different outcomes.

Penetration testing

The most commonly referenced type of security testing is “penetration testing.” That has become a catchall term, and, unfortunately, it’s misleading. 

True penetration testing is a tactical service suitable for robust, hardened, thoroughly tested systems. It’s a time-constrained effort to measure a single outcome. For example, a penetration test might seek to determine “could an attacker escalate basic user privileges to admin rights?” The outcome is either yes or no. There is no other outcome.

Penetration testing is great when you have a mature, well-tested defense, and you want to determine if that defense still stands up to a simulated attack. Think of it like when a car maker wants to understand how a vehicle performs in a crash situation. They literally crash it into a wall to see what happens. It’s great for understanding a specific scenario. But it is only good for a system that’s already been thoroughly tested. And it’s really only intended to inform what happens in that specific scenario; it’s not intended to be holistic.

Vulnerability scanning

Now, that’s what penetration testing is supposed to be. It’s badass, and it seeks to find exploits. Unfortunately, the term is often used to refer to something else: vulnerability scanning. Thanks to misleading marketing and confused customers, “vulnerability scanning” has become synonymous with “penetration testing.” 

It’s not.

Vulnerability scanning involves running an automated tool that looks for common vulnerabilities that are known to exist. The goal is to quickly and inexpensively find basic issues, including checking for unpatched vulnerabilities. Given that running a scanner is one of the first steps your attacker will take, it’s a good idea for you to do this too. You want to see what they see. It’s good if you want to keep timelines and cost to a minimum (with the understanding that it will also limit the value you get as a result).

It’s like the diagnostic tool that mechanics use when the “check engine” light comes on in your car. The tool scans for known issues, spitting back readable codes. It’s easy, inexpensive and quick. But it’s certainly not a comprehensive way to evaluate vehicle safety.

Think about that: you ask to simulate a car crash, yet are sold a way to remove the check engine light. Those are pretty different!

Vulnerability assessments

The frustrating confusion doesn’t end there.

As if asking for one thing but being sold something else wasn’t frustrating enough, there’s this: neither of those deliver the outcome that people are usually after. 

When it comes to security testing, most people are looking for a comprehensive understanding of their system’s security vulnerabilities. They want to know what the problems are across the entire system and how to fix the problems, with a way to prioritize what to focus on first. Then they want to be able to prove that the system is more secure.

That’s not what either penetration testing or vulnerability scanning delivers. But it’s exactly what vulnerability assessments deliver. 

Vulnerability assessments leverage experienced humans who solve problems manually in order to address your unique circumstances. In the real world, you’re defending against smart, motivated, problem-solving humans — not just scans. Vulnerability assessments help you defend accordingly. They’re great for both well-hardened systems as well as those still figuring it out (and everyone in between). 

Unlike a single crash test or running the diagnostic tool to clear the “check engine” light, vulnerability assessments are like the entire safety engineering department. They consider all of the different safety systems — from seatbelts to airbags — and how they all work together. It’s a holistic view on where the weaknesses are and how to improve them.

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

Know your goal

As you can see, each of these terms means something quite different, so it’s important to understand four things. First, there is a difference. Second, terms are commonly misused for each other. Third, they shouldn't be. Fourth, it’s up to you to make sure you get what you need. 

The best way to do this is to start with your goal. What do you want to achieve with the testing?

If you have a mature, heavily hardened system that’s already been through extensive security testing and you want to know how it stands up to a simulated attack against a specific area, get penetration testing.

If you need to find basic, common issues quickly, keep costs to a minimum and are fine without finding custom exploits, get vulnerability scanning.

If you need to find as many security vulnerabilities as possible — including custom exploits — understand their severity, and fix them based on priority ranking, get vulnerability assessments. Be clear on your goal when discussing testing with your security company, and you’ll be able to get the outcomes you need, irrespective of which term is being used to refer to the testing.