How to Run a Successful Proof of Value for an Application Security Programme

So you’ve got upper management buy-in for your application security proof of value and are ready to start scanning applications: how do you make sure your proof of value (PoV) is a success and that you demonstrate the need to progress to a full-scale program? This article describes some of the lessons learned at the start of our large-scale deployment of Veracode within our organisation.

Socialising the Proof of Value

The first step is to socialise the PoV internally through word of mouth, discussion forums and developer communities by driving interest in the availability of a new tool for developers, which will assist in the development process and produce better code. Ensure that you are familiar with the platform and the various IDE plugins and can demonstrate their effectiveness on a real-world application (we used the OWASP WebGoat application as our technology demonstrator). The emphasis should be proactive use of the tool to detect flaws at the point of introduction rather than as a security measurement tool. Key success factors for development teams will be the integration of the tool within common IDEs and the ease of adoption (specifically no need for in-depth product knowledge or the use of vendor specialists). Once you are familiar with the platform and your toolbox, you will need some applications to scan.

Application Selection

The selection of which applications to scan is a key success factor: it is important that the applications chosen have strategic significance to the organisation in order to demonstrate the significance of the findings to senior stakeholders. Much of our difficulty arose in determining suitable applications since this information may not be readily presented in an application repository (and an application repository may be inaccurate or not exist at all). The forging of informal networks and word of mouth will be essential to success – talk to people in the canteen, and be on the lookout for internal events of interest to developers; in our case, we had access to an internal social media site, which was an excellent platform for creating interest and awareness. Do resist the temptation to scan applications that are of low importance simply because they happen to be available; this will reduce the impact of your PoV.

Building and Scanning

Now for the moment of truth: building the code and performing the scans. It is vital that you or your team have a working knowledge of the mechanics of building software – getting access to source code is one thing, but expecting a busy development team to help you perform the requisite debug builds for scanning is unlikely to be met favourably. The ability to speak the developer’s language is a key objective in establishing an application security programme, and demonstrating our competence with their environments and toolchains gave us credibility when conducting the initial reviews of the scan results. Be sure to review the findings internally before distributing to the teams, and ensure that your team is familiar with the nature of the findings and can speak confidently to the risk presented by such flaws. Establishing and maintaining the credibility of your team is vital at this stage. By this stage, there is certain to be a high level of interest in the PoV from within various parts of your organisation, and it is important to demonstrate results as soon as possible. Indeed, many teams will be eager to see their application’s results. Be sure to manage expectations around expected scan times from Veracode to avoid any possible negative perceptions of the use of a SaaS product; the emphasis should be on the ease of adoption and lack of specialised knowledge.

Common Criticisms

You should be prepared for a fair deal of scepticism around your initiative, which may be based on outdated views of the capability of static code analysis tools, through to a belief that no problems exist within the organisation. A common objection to the use of static code analysis is the high false positive count or lack of actionable output; ensure that all applications scanned in the PoV have a consultation call to demonstrate the accuracy of the findings and the specificity of the results in the flaw viewer. Negative feedback from a development team to their management could be disastrous for your future program. At the conclusion of the PoV, you will need to demonstrate the value of your PoV to senior stakeholders: the key message is the rate of flaw detection that has been achieved, emphasising that such rates could not have been achieved by any manual review process. In our case, we demonstrated a comparison between a Veracode scan and the traditional approach of an application penetration test; the benefits in terms of cost (in our case by an order of magnitude) and timescales clearly favouring the Veracode review tools. Identifying the so-called “smoking gun” is useful in demonstrating the need for a larger-scale programme; however, be sensitive to the application team concerned and emphasise the role of your team in an advisory capacity in reducing vulnerabilities. By now, you will have clearer view of the application estate within your organisation and will have an appreciation for the challenges you will face in scaling this process to a fully-fledged programme.

For more details on my experience developing an application security programme, check out my new guide, From Ad Hoc to Advanced Application Security: Your Path to a Mature AppSec Program.