Deciding to outsource application security testing to a managed service provider is one step toward simplifying your AppSec testing strategy and improving the impact of test results.
But simply choosing a provider and getting outsourced testing up and running doesn’t mean you’re done. It’s also critical to measure how well the managed testing service performs so that you can validate whether it’s doing what you need it to do.
With that need in mind, let’s take a look at which KPIs, metrics, and other types of measurements you should consider when assessing the value of managed AppSec testing. Although every business is unique and your mileage may vary, the data points we describe below help form the foundation for effective measurement of outsourced AppSec testing.
The key KPIs you’ll want to measure can be broken down into three main categories: security impact, productivity impact, and cost impact.
On the security front, consider measuring KPIs and metrics such as:
- Total vulnerabilities discovered: This metric, which tracks the total number of vulnerabilities your AppSec testing provider identifies, provides a holistic overview of the effectiveness of the testing process. Ideally, outsourced testing will reveal more vulnerabilities than you were detecting using in-house testing.
- Vulnerabilities discovered in source code: In addition to measuring total vulnerabilities detected, track how many the provider discovers in your source code – as opposed to after your application is built. Shift-left security is all about detecting risks as early as possible in the development process, so vulnerabilities detected early on are especially valuable.
- Vulnerabilities discovered in production apps: On the other side of the coin, measure how many vulnerabilities aren’t detected until your application is in production. Ideally, this number will be zero – but if it’s high (or higher than what you were achieving using in-house testing), it’s a sign that your outsourced AppSec testing service is not meeting the number-one goal, which is to minimize the number of security risks that make it into production environments.
To contextualize these metrics further, consider collecting them separately for each application or microservice you develop, if possible. That way, you’ll have granular visibility into how outsourced security testing is or isn’t improving the security outcomes for each part of your codebase.
In addition to enhancing security, one additional major motivation for outsourcing security testing is to allow your in-house staff to focus on other work. For that reason, you should be monitoring the extent to which your development and IT teams become more productive after you outsource testing.
You can track their productivity using metrics such as:
- Total code commits: Ideally, your developers will be able to write more code – which you can measure by tracking code commits – when you outsource testing.
- Application release frequency: Your CI/CD pipeline should also be able to move faster and generate more frequent application releases when you are no longer distracting your developers with AppSec tests.
- Time-to-provision new infrastructure: Although measuring the productivity of IT engineers can be a bit tricky (because IT engineers do so many different things), one useful metric to track is how long it takes them to stand up new infrastructure. If outsourced AppSec testing is effective in reducing the burden placed on your in-house teams, your IT engineers will be able to set up new infrastructure more quickly than they were before you outsourced your testing.
You can measure these KPIs more granularly if you break them down on a team-by-team basis. That way, you know which specific groups of developers or engineers benefit most from outsourced testing.
Reducing overall testing costs is another common reason for outsourcing AppSec testing to a managed service provider. To determine how much you’re saving, measure:
- Cost per vulnerability: How much do you pay on average to detect each vulnerability? As an example, if you spend $1000 per month on testing and you detect 100 vulnerabilities per month, then you’re paying $10 per vulnerability. Ideally, this number will be lower than what you were paying in salaries to your engineers to discover vulnerabilities using in-house testing.
- Salary spend: You can also track your total spending on developer and engineer salaries. Although many factors other than security testing workloads can affect salary costs, it’s at least helpful to know whether your salary spend is trending upwards or downwards after you invest in outsourced testing.
Both of these metrics may be a little imprecise due to the many variables at play in calculating spending. But even if the cost data is not perfectly clear-cut, it can nonetheless provide a general sense of whether outsourced AppSec testing is paying off economically.
Again, the right KPIs to measure for your business depend on the exact nature of your teams and engineering practice. But in general, you’ll want to focus on collecting and assessing metrics that help you understand how much outsourced testing improves application security, makes your engineers more productive, and saves your business money. When you can quantify these factors, you can demonstrate clearly whether, and to what extent, an outsourced testing program has a net positive impact on your business.
Chris Tozzi has worked as a Linux systems administrator and freelance writer with more than ten years of experience covering the tech industry, especially open source, DevOps, cloud native and security. He also teaches courses on the history and culture of technology at a major university in upstate New York.