EVERYTHING You Need to Know About

Measuring Your
APPSEC PROGRAM

By the Numbers

Before a company begins a new project or program, it’s important to determine what success for that project means. Setting metrics and goals based on the program’s objective can help companies and departments evaluate the success of a program or provide insight into where adjustments should be made. While metrics and key performance indicators are at the center of a performance-based culture, many organizations continue to miss the mark by measuring the wrong things or focusing on aspects of a program that don’t truly measure its health.

The stakes of measuring the wrong aspects of your program are even greater when it comes to application security. Amid a fast-changing and increasingly perilous landscape, a lack of visibility and understanding can undermine or even cripple your enterprise, and open the door to breaches, breakdowns, and stolen data — all while you believe your program is working because your KPIs indicate success.

Metrics — or perhaps more accurately, the right metrics — are crucial for understanding what’s really happening in your AppSec program. They serve a dual purpose: They demonstrate where your organization is at but also show what progress it’s making in achieving its objectives.

TweetThe key to a truly effective AppSec program is to identify the right metrics and present them in a way that your executives and key business decision-makers can understand and use. An overly cumbersome approach, or one that presents information in an obtuse and burdensome way, isn’t likely to succeed.

This means devising a program that supplies the right metrics — and in some cases the right single metric — to a decision-maker at the specifc time and place it’s needed. For most companies, this task revolves around two overarching factors: the total number of apps in your program and the percentage of apps that are in compliance with your AppSec policy. Together, these criteria allow your organization to drill down and understand an array of factors and conditions that can make or break your cybersecurity initiative.

ANNE NIELSEN

Measuring policy compliance is a critical aspect of an effective application security program.
It’s key to focusing your investments and resources, quantifying your progress, and understanding your risk profile.”

ANNE NIELSEN
Principal Product Manager, Reporting and Analytics, Veracode

THE CASE FOR METRICS

Application security presents a few unique challenges. First, today’s software landscape — from the products themselves to development methodologies to the types of threats — is constantly changing and evolving. Add in DevSecOps and other Agile initiatives, and the challenges grow, sometimes exponentially. Second, the number of applications and the volume of code that organizations must manage are continually growing. All of this introduces new and different threats, particularly as organizations increasingly take advantage of open source libraries to speed their software development. Third, amid an overwhelming barrage of cybersecurity threats, AppSec is only part of the picture — it’s just one piece of your overall cybersecurity program.

Yet, application security doesn’t have to be an overly complicated or burdensome task. The right set of metrics and key performance indicators (KPIs) can greatly simplify and streamline both your software development and your application security. SANS Institute points out that AppSec metrics fall into three major categories: technical, operational, and executive.1 TweetOrganizations that use the right AppSec metrics and make them available to the right people channel resources more effectively and improve code quality. This has positive repercussions beyond the four walls of your organization: It reflects outward to your business partners and customers, who make decisions based on the quality and security of your code.

1 Benchmarking AppSec: A Metrics Pyramid. SANS Institute

GUIDE
By using metrics and tangible data to support an application security initiative, you can take performance to a higher level. Find out how by reading Building a Business Case for Expanding Your AppSec Program.

MEASURES OF SUCCESS

Establishing a metrics framework that delivers best-practice results starts with recognizing what needs to be measured. Ultimately, an effective AppSec program revolves around four crucial metrics: policy compliance, scan activity, flaw density, and fix rate. Let’s take a brief look at each of these factors and why they’re essential:


METRIC1

Policy Compliance

Your organization likely conducts threat modeling and risk ranking to understand where to focus scarce security resources. Your application security policy should stem from a similar exercise, starting with looking at your entire application inventory and assigning groups of applications different risk categories. Develop these categories by asking simple questions such as:

  • Do these applications touch PII?
  • Are they Internet-facing?
  • What would be the impact of a compromise to this system (i.e., are they business critical)?

Based on those answers, you can determine which scan frequency and testing types are required, as well as which types or severities of flaws may be acceptable. Perhaps disallowing the OWASP Top 10 is sufficient to meet your security needs, or maybe you want to disallow the SANS 25 or all very high severity findings. You should also be stipulating the appropriate testing techniques for each grouping of applications. Internet-facing applications likely require dynamic analysis in addition to static analysis. Also consider the frequency of testing for each group of applications. Perhaps an annual manual penetration test is required, both to meet your PCI requirements as well as your risk ranking.

It's important to note that you shouldn’t develop the security policy for your organization in a vacuum, but rather in conjunction with software development. Working with development teams to ensure that the security policy is both addressing the risk for the organization and achievable will help ensure adoption overall.

  • Conduct threat modelling/risk assessments on your application inventory and build risk tiers.
  • Determine security policy based on the tier, including the testing techniques and frequency, in addition to disallowing specific findings types, severities, or categories.
  • Work with software development to assess all applications, starting with your most sensitive tier.
  • Partner with development teams to understand their needs in order to meet your security policy compliance requirements. Perhaps they need access to additional AppSec training or expertise to determine if an external compensating control is effective at removing risk associated with a security findings, or if remediation (a code change) is required to address the security finding.

METRIC2

Scan Activity

Organizations, particularly as they move to Agile and DevSecOps, should find themselves scanning applications and code more frequently. And it’s critical for these scans to occur at the right time and within the right workflow that aligns best with the development team. When scans are out of alignment with processes, the risk of introducing vulnerabilities increases. Critical metrics revolve around tracking when scanning is occurring: Is it with each release or on a schedule (daily, monthly, yearly)? In addition, is scanning taking place in an ad hoc fashion or through automated integration with development systems?

  • Engage development teams and other key stakeholders on the release process and determine when to best integrate security testing. Based on the release cycle, testing may occur with each individual developer’s branch or at the end of a development sprint as part of a release gate — or anywhere and everywhere in between.
  • Train development teams on their options for integration points for security testing. While the security team sets the requirement for security testing through the security policy, individual teams should own the way they meet that policy.

METRIC3

Flaw Density

Often, the applications for different development teams, business units, or groups need to be compared in order to understand overall risk and focus application security resources in the most impactful location. However, comparing applications can be challenging. Policy compliance as a metric may help here, but if both groups of applications are failing policy, where is the greater risk?

Further, the prevalent categories or types of flaws differ from language to language; a MicroService written last week that’s 1MB differs greatly from a legacy stem written fve years ago. So how do you compare apples and elephants? Flaw density provides a way of looking at the number of flaws produced from a static analysis over the size of the application and can provide directional guidance when comparing groups of applications. A high flaw density simply means more flaws to address. You could choose to focus only on very high and high severity flaws for your flaw density metrics and better compare sets of applications. This comparison would give managers a sense of where to use application security resources — such as training or working with remediation consultants — to have the biggest impact for your business.

  • Focus on flaw density to better compare platforms, systems, development languages, and apps.
  • It’s also wise to view this data over time to better understand trends and the effectiveness of remediation methods.

METRIC4

Fix Rate (Time to Fix)

TweetAn AppSec program is only as good as the flaws it actually fixes. If your development teams are addressing fixes quickly and effectively, your organization’s risk exposure is likely to shrink. If the fix rate falls short, however, your development teams may need to adjust. For instance, your organization may require new or better training methods, or it may need to adapt workflows and processes to better address deficiencies. For open flaws, it’s important to understand your “high” and “very high” severity findings. This will allow your organization to direct resources to where they’re most urgently needed.

  • Track your closed findings — both those that have a documented compensating control (mitigated) and those that required a code change (remediated) — to understand where you are making progress.
  • Are specifc categories remaining open longer? Perhaps training on addressing those categories or CWEs is needed.
  • A Dynamic Vulnerability Rescan tracks and catalogs flaws by key categories, including “new,” “fixed,” “opened,” and “reopened,” since the last application scan took place. This tool can greatly simplify checks on code and provide data about what’s working effectively and what needs to be addressed.
  • Provide education, training, eLearning, and consultation calls for developers and other key constituents as they’re needed. Aid developers in understanding the value of the metrics you’re using and how tools can help your enterprise achieve its AppSec objectives.

WEBINAR
Learn more about the metrics that can help you prove your applications are secure and that your AppSec program is making a difference in our webinar, “Application Security Metrics & How to Track Success.”

HOW TO BUILD A ROBUST METRICS FRAMEWORK

Integrating metrics into workflows and processes is critical. According to OWASP, organizations must address several important questions in order to build a strong metrics framework.2 These questions revolve around three areas: application security process metrics, application security risk metrics, and software development lifecycle (SDLC) security metrics.

Application security process metrics
  • How  well is our organization meeting security policies, technical standards, and industry practices? How consistently are we executing security SLAs? By application? By division? By channel?
Application security risk metrics
These encompass vulnerability risk management metrics:
  • What is the mean time to repair on an annual basis? On a monthly basis? By application? By division? What are the known security issues in production?
They also include security incident metrics:
  • What security issues have been exploited? Were they known issues that were released in production? What was the cost to our business?
They also include threat intelligence reporting and attack monitoring metrics:
  • Which applications are receiving more attacks than others? Which applications have upcoming expected peak usage?
2 CISO AppSec Guide: Metrics for Managing Risks & Application Security Investments. Part IV: Metrics for Managing Risks & Application Security Investments.
SDLC metrics
These include metrics for risk mitigation decisions:
  • What is the mean time to repair by an application's risk category? Does it meet expectations? What is the risk heat map by application? By division? By channel?
They also touch metrics for vulnerability root causes identification:
  • What are the root causes of vulnerabilities for each application? Is there a systemic issue? Which security practices have been best adopted by each development team? Which development teams need more attention?
And, fnally, they encompass metrics for software security investments:
  • Which SDLC phase has identified the most security issues? What is the maturity of the corresponding security practices in each SDLC phase? What is the urgency for more security people, process, and technologies in each SDLC phase?

ANALYST REPORT
Get details on integrating security into the software development lifecycle in this Securosis report, Building an Enterprise DevSecOps Program.

A METRICS-CENTRIC CULTURE IS PARAMOUNT

Metrics are more than a collection of numbers. They represent underlying business requirements and organizational objectives that lead to better performance. In fact, the business context behind the numbers is just as important as the numbers themselves. It’s easy to overlook this fact in the daily crush to complete projects. TweetMetrics are useful only when your organization has a culture that recognizes the importance of KPIs and has a framework in place to support crucial metrics. There are four best practices for building metrics into the fabric of your enterprise:

Executives need to understand the story behind the metrics. It’s a basic truth: No enterprise initiative succeeds without executive buy-in. However, cybersecurity and AppSec are particularly challenging tasks. For them to succeed, thought leadership and messaging are crucial; funding for the right tools, systems, and technologies is critical. Organizations achieve best-practice results when they deliver the right level of information in the right format to executives — while providing your executives and development teams with more detailed metrics, as appropriate. The executive team needs to understand the metrics being used and how they’re related to success and reduced risk.

Developers must understand why metrics matter. It’s logical to think that your developers should understand the need for metrics and key performance indicators. But, historically, cybersecurity and application security haven’t been the primary focus for developers. Writing code and producing applications is at the center of their world. What’s more, as organizations shift toward DevOps, there’s greater pressure to work faster. The upshot? Your organization must have a framework in place that focuses on a few key issues: educating developers about how code quality and application security are deeply intertwined; keeping them informed about vulnerabilities, typically through the OWASP Top 10; and providing incentives and rewards for improving application security and code quality.

The right metrics must be supported by the right data. It’s ironic that an organization can hit all of its metrics goals and still fall short of its performance objectives. This problem occurs because of two possible scenarios: The organization establishes the wrong metrics but succeeds in achieving these metrics, or the underlying data measurement or collection system is flawed. Either way, the enterprise winds up working with the wrong data and, in AppSec, channeling resources inefficiently. Mapping out high-level metrics and secondary metrics that support the primary goals of your organization is critical. After all, not all vulnerabilities are created equal. For AppSec, this means understanding what truly represents the greatest risk to your enterprise and addressing these concerns accordingly.

The right policies must be in place to support critical metrics. In order to ensure that metrics and data are in lock-step, there’s a need for discussion that involves key executives, line of business leaders, and managers from development teams. It’s critical to create a framework with the right set of internal guidelines, standards, and policies to support an AppSec initiative. And this framework should be based on your threat modeling and the risk ranking of your application inventory. This allows your enterprise to create workflows and processes that support best-practice application security. Yet, it’s also important to have tools and systems in place to track problems, manage tasks, and address priorities. This includes tools for scanning code and tracking adherence to policies. As external compliance requirements grow through regulations like the EU GDPR and the NY DFS’ cybersecurity regulations, your organization must also factor these requirements into the AppSec equation.

METRICS = BEST PRACTICES

A holistic and comprehensive application security program revolves around both high-level metrics and supporting metrics. It involves also people, processes, and technology. When all these factors are aligned — and when executives, line of business managers, and software developers are connected and tuned into the specific requirements of the organization — AppSec evolves from a general concept to a powerful tool. A metrics-centric AppSec program is particularly valuable in today’s business environment, where Agile, lean, and DevSecOps rule. A core tenant of each of these is to measure/make work visible. In order for application security to truly be embraced by software development, follow this philosophy and provide clear visibility into how the program is running. Ultimately, the right metrics can help your organization reduce risk but also achieve a competitive advantage through higher-quality software code. The right metrics also reduce costs associated with security and aid in an organization becoming more proactive. This, in the end, generates both top-line and bottom-line gains.



We’ve helped thousands of companies, large and small, develop AppSec programs that work for them, and we can help you.

Contact Us for help getting started or for moving to the next step.

ABOUT VERACODE

Veracode is the leading AppSec partner for creating secure software, reducing the risk of security breach and increasing security and development teams’ productivity. As a result, companies using Veracode can move their business, and the world, forward. With its combination of automation, integrations, process, and speed, Veracode helps companies get accurate and reliable results to focus their efforts on fixing, not just finding, potential vulnerabilities. Veracode serves more than 2,500 customers worldwide across a wide range of industries. The Veracode cloud platform has assessed more than 14 trillion lines of code and helped companies fix more than 46 million security flaws. Learn more at veracode.com, on the Veracode Blog, and on Twitter.

Copyright © 2021 Veracode. All rights reserved. All other brand names, product names, or trademarks belong to their respective holders.