What Gets Measured Gets Done: a Motto to Live by in Application Security

Back in December, the CISO of a financial services company explained how he took his company’s application security program from 0-60 in 12 months. Now, that same CISO explains why measurement was a critical component to the program’s success.

As we developed our application security strategy, gaining buy-in from various stakeholders was an essential part of making it a success. But, we also knew that we’d need to standardize on a set of effectiveness measurement so that we could measure progress and demonstrate that we were reducing risk. Also, I live by the motto “what gets measured gets done”, meaning if we create numerical goals, we are more likely to work toward reaching the goal.

Once we determined with our stakeholders what our application security program goals were, we created a dashboard of AppSec measures. This quantified the effectiveness of each control we put into place. The highly visible dashboards held us all accountable for the changes we were making, and promoted adoption of new controls across all teams.

flaws fixed is a positive metric – it reports on the improvements and progress the development team is making

For example, one of our key initiatives was training our development team on secure coding practices. We knew that doing so would ultimately improve our overall security by reducing vulnerabilities before we even assessed applications. However, we also knew that this is a hurdle many application security programs don’t overcome. Developers are on tight deadlines to produce innovations. From the developer perspective, training programs simply get in the way of productivity and on-time releases. But, by setting up a dashboard measuring training sessions completed on time, we are communicating that training is a priority, not a luxury. 

Many application security programs measure the number of vulnerabilities found. Though this can be a helpful metric for baselining risk, we felt creating a dashboard measuring flaws fixed within an agreed upon SLA was also a worthwhile endeavor. The flaws fixed metric is important because it shows progress toward reducing risk, but it also has a psychological and collaborative benefit. Measuring vulnerabilities found can cause developers to feel like they are being shamed or criticized. After all, you are basically finding all their mistakes and reporting on them. But flaws fixed is a positive metric – it reports on the improvements and progress the development team is making. As a result, our security team is able to have a more cordial relationship with the development team.

Of course, these are not the only two metrics we reported, we had 1-3 measures established for all of our secure SDLC controls. But they are good examples of how we used metrics to build, refine and iterate our program. Now that we have a solid program in place, we can use these and other metrics to demonstrate the value of the program and explain how expanding the program to cover more of our application portfolio can benefit the company.