Metrics — or perhaps more accurately, the right metrics — are crucial for understanding what’s really happening in your AppSec program. They serve a dual purpose: They demonstrate your organization’s current state, and also show what progress it’s making in achieving its objectives.
We typically recommend our customers measure their compliance against their own internal AppSec policy, plus scan activity, flaw prevalence, and time to resolve.
Flaw rate is another metric you might want to consider tracking. Although this would be a secondary metric, unlike the primary ones listed above, flaw rate, which allows you to do a before-and-after flaw comparison for an application, provides insight into how your rate of security findings is improving over time. Veracode analytics allows you to create the flaw rate metric by using a formula and adding it to your chart in order to visualize the rate alongside any other data you are reporting – such as flaw rate per application, first scan vs most recent scan, or flaw rate per an application per severity of the finding.
Keep in mind that this metric, as with flaws per MB, can vary significantly based on the size of the codebase. A monolithic, legacy application is going to have a much different flaw rate (and flaw density as measured by flaws per MB) than a small, new microservice. The value lies in comparing an application’s initial flaw rate to the current flaw rate, or comparing the flaw rate for a team across several applications (again the initial flaw rate vs. the current). This allows users to get a handle on what is working – or not – for that team to help them close out security findings and reduce the number they are introducing in the first place. In this way, you could validate the impact of your AppSec eLearning or other trainings. I would caution against comparing flaw rate (again much like flaws per MB) between teams or between business units as this won’t directly provide much actionable insights beyond which one is doing better.
Note that this metric will not produce an accurate gauge of your program’s success. Since it is applicable only to static analysis, it doesn’t take all testing techniques into account. Policy compliance is ultimately the best metric for measuring and reporting on the overall progress of your program.
But you could use flaw rate as an additional data point, alongside the following metrics, when reporting on the effectiveness or progress of your AppSec program:
Policy compliance: Your application security policy should stem from an analysis of your entire application inventory. From there, you assign groups of applications different risk categories or ratings by asking questions such as:
- Do these applications touch PII?
- Are they Internet-facing?
- What would be the impact of a compromise to this system (i.e., are they business critical)?
Based on those answers, you can determine which scan frequency and testing types are required, as well as which types or severities of flaws to disallow: an Internet-facing application that contains PII will have a different risk categorization from an internal chat service and thus should be held to a different standard for security.
Additionally, this risk rating will determine frequency of scanning requirements. Low-risk functionality that is rarely updated does not need to be scanned every week, but that Internet-facing/PII app may require a scan for every commit.
Average time to resolve: Many application testing solutions focus on scan activity rather than addressing results. While apps need to be scanned, fixing those security findings in a timely manner is a better mechanism for evaluating your application security program. Time to resolve provides visibility into how many days it takes for a finding to be closed after it is first discovered, helping security teams better understand where there may be bottlenecks in the development and security process.
Flaw prevalence: This metric spotlights how common a risk is within a particular industry or business. It helps an organization prioritize threats such as SQL injection, Cross-Site Scripting (XSS), cryptographic issues, and CRLF injection based on real-world impact.
Learn more about flaw rate
For detailed instructions on measuring flaw rate, please see this article in the Veracode Community.
