How to Evaluate and Select Application Security Testing Vendors

The application security testing (AST) market is getting crowded. In addition, many of the vendors offer multiple technologies, and are promising even more advanced technologies in the near future. Some deliver technologies as tools, some as services. And these technologies are all priced differently. The question is: How do you evaluate the marketplace and select the right vendor?

In this blog, we offer a decision framework that is based on evaluating vendors along two major sets of criteria: technology and business. This framework gives you all you need to know about the vendors and the market, in order to make an optimal vendor selection.

Technology

Breadth of AST technologies

No single technology can provide complete insight into an application’s security. Therefore, an optimal vendor should offer more than one of the following technologies and features:

• Static application security testing (SAST), which analyzes code for security vulnerabilities early in the lifecycle, enabling the least expensive and fastest remediation.
• Software composition analysis (SCA), which detects third-party (mostly open-source) software components with publically known vulnerabilities.
• Dynamic application security testing (DAST), which analyzes application behavior at test-runtime under simulated attacks and, based on that, points to the detected vulnerabilities.
• Third-party vendor software testing: This is an ability to analyze third-party packaged applications. This process typically involves one or more technologies such as SAST, DAST and SCA.
• Application discovery: This is an ability to analyze an enterprise’s web ecosystem and detect all web applications. Often, it discovers a substantial number of applications that enterprises do not use and had forgotten about, and yet still give cyberattackers a gateway into the enterprise.
• Enterprise-class capabilities: These enable the integration of AST technologies into IDEs, bug-tracking systems and code management systems. They also enable integration with SIEM and GRC systems for analytics, reporting and result consolidation.

AST delivery models

Delivery models come in two main variations:

• Tools
• Cloud services

When enterprises select tools as an AST delivery model, they have to install those tools, maintain them, train employees to operate them (or hire experienced specialists), and be responsible for the results of the tests. Typically, that delivery model does not scale, is expensive, and requires skills and time (which are in short supply) to operate AST tools. Cloud services, on the other hand, do not require that enterprises buy tools, install them, maintain them, learn how to use them, run them or take responsibility for the accuracy of vulnerability detection or the latency between test request and results return. All these tasks are handled by third-party independent experts on behalf of the enterprise. AST-as-a-cloud service is an advanced delivery model that makes security transparent to development and operating specialists.

AppSec management

AppSec management enables enterprises to run their AST effectively by providing:

• Remediation support: The AST vendor should help developers understand vulnerabilities detected by the AST technology and help effectively fix them.
• Help maturing the AppSec program: The AST vendor should be helping enterprises to evolve AppSec from casual use to a rigorous, planned and measured program.
• Training: The AST vendor should be able to provide AppSec classes on security awareness, types of vulnerabilities, and secure coding best practices.
• Technical support: The AST vendor should respond in a timely and effective manner to the concerns of its clients. Tech support can be via phone, electronic media (i.e., WebEx) or sometimes on-site. It can be provided round-the-clock, or during the most pressing business hours.

Innovation

There are several AppSec areas where technology innovation and thought leadership are most required. Among them are the following:

• Securing DevOps: Find out if the vendor is capable of offering AST technologies that can be applied incrementally, to the smallest snippets of the code, that AST is transparent and does not distract Dev and Ops specialists from doing their jobs, that the request for AST can be submitted automatically, and results returned with minimal delays.
• Beyond testing: application protection – Find out if the AST vendor offers either application protection technology (e.g., RASP – runtime application self-protection) or can integrate with other vendors’ WAFs. A vendor’s ability to detect vulnerabilities and protect applications against attacks makes its application security offering more complete.
• Beyond web testing: testing mobile applications – Find out if an AST vendor can test not only web applications, but mobile apps as well. Mobile app development is rapidly evolving, and security testing of the mobile app code, mobile app behavioral analysis, and security analysis of communications between mobile apps and enterprise-class web applications are becoming important trends.

Business  

The business component of an AST vendor evaluation addresses the vendor’s viability.

Is your vendor doing well? You must evaluate the following:

• Is the vendor financially stable? Will it exist in the market in the years to come?
• Get proof that the company will keep investing in AppSec. Some vendors change their directions, stop investing in some of their products, and start investing in other products that aren’t relevant for your organization.
• Changes in executive team, talent attrition: People make the company. Review recent changes in the vendor’s human resources. Be cautious if the vendor has lost its leading innovators and decision-makers.   

What is the vendor's marketshare? Evaluate the following vendor characteristics:

• Revenue
• Number of enterprise clients
• Number of individual users

Consider mindshare. Evaluate the following vendor characteristics:

• Vendor reputation: Does the market and the user audience respect this vendor?
• Market awareness: Is this vendor well known?
• Track record: How successful has this vendor been through its history?
• Ability to change directions when new trends evolve and when the market changes.
• Innovation: Is the vendor capable of innovating? How transformational has its innovation been?
• Appeal to a broader audience: Does the vendor’s solution appeal to not only security specialists, but also to developers, QA and testing specialists?

Finally, price is an important consideration.

• What is the price per unit, and what is the unit: tool, scan, etc.?
• Can the price model scale, but not become too expensive?
• Do you have choices to combine different pricing models?
• Do you understand the price arithmetic?

Summary

Look for a suite of technologies. For a vendor that offers multiple technologies:

• Look beyond a single point solution.
• Look for multiple testing technologies such as SAST, SCA and DAST.

Give preference to vendors that can offer not only detection, but protection as well. In addition, seek out a vendor that enables testing of third-party software:

• Custom packages
• Open-source software

Look for cloud services provider, especially if you lack AppSec skills/resources. And ensure that your vendor is financially viable, stable and reputable. If you are happy with all of the above, negotiate the price.