A Few of My Lessons Learned Building an AppSec Program

I recently joined Veracode after spending five years building an application security program from the ground up at a global investment bank. This experience gives me a unique perspective on the struggles and hurdles our customers are facing, and puts me in a position to share my lessons learned and provide helpful information and advice for those starting or managing a growing application security program.

During my time at the bank, our path to an advanced application security program followed three major steps: Define/Discover, Execute and Optimize. The following is a sampling of the key takeaways and lessons learned from each of those steps.

Lessons Learned in Define/Discover Step

A couple of early steps in our application security program that contributed to our later success included:

Getting stakeholder buy-in: Without this early buy-in, we would have struggled to see results from our program. Application security is not a solitary pursuit; it’s a far-reaching initiative that affects the processes and priorities of a variety of different individuals and groups in your organization. Without getting these stakeholders on board early, your program will never get off the ground. We started the process of getting stakeholders on board by:

• Identifying the business owners of affected applications.
• Identifying application and technical leads.
• Socializing the program on internal websites.
• Holding kick-off sessions in key locations.

Conducting an asset inventory: Beginning our program without first getting a clear picture of our application landscape and its dependencies would have been inefficient, ineffective and eventually insecure. We began constructing an application inventory by:

• Determining how the organization kept track of its application assets (who owns the apps, what is their business unit/owner/criticality/regulatory requirements, etc.).
• Getting access to software source control systems.
• Understanding business criticality classification.
• Aligning with risk functions to agree on scoping.

A couple steps we didn’t take early on in our AppSec program, which made our job more difficult later, included:

• Securing a mandate: We should have done this earlier. In the first three years we relied on the cooperation of development teams to participate; they could have easily refused, and we would have had no rights to enforce participation.
• Allowing application teams to be de-scoped too easily: Once applications were excluded from the program, it was hard to bring them back into scope.

Lessons Learned in Execute Step

As our program expanded, one key step that kept the program moving forward smoothly was to set the application security policy to a very achievable level: no XSS, SQLi or CMDi.

At Veracode, we work with our customers to set application security policy and always recommend not to set the bar too high. While a high standard of security is vital, unrealistic expectations and requirements too often lead to people looking for ways to circumvent policies so they can get their work done. A basic fact of application security is that any policy should be only as complicated as it needs to be to deliver the necessary results. A more realistic approach starts with attainable goals — such as the eradication of a specific high-risk threat — and expands from there.

However, although our policy was set to a manageable level, we had poor governance over the scanning process. We were missing a feedback loop that would ensure developers were adhering to policy and that we were maintaining governance of the process (ensuring that developers could not bypass our assessments).

Lessons Learned in Optimize Step

In this step, we focused heavily on remediating found vulnerabilities, and also incorporating security testing of third-party applications and open source components.

A big lesson learned in this phase was the importance of creating and nurturing security champions. We learned that the best way to create these champions was to:

• Coach developers who self-identify as security experts.
• Incentivize within teams to encourage security champions.

At this point, we also encouraged development teams to use Veracode’s Developer Sandbox and perform rescans. In this way, they were free to scan and rescan their code without failing policy and adversely affecting security dashboards and business KPIs.

Finally, as we got deeper into remediating or mitigating vulnerabilities, we found success by developing a standard procedure for proposing, documenting and processing mitigations. This procedure required that each mitigation proposal include the following documentation:

• Technique: Type of mitigation in effect
• Specifics: Specific compensating control in effect
• Risk: Risk that mitigation does not address
• Verification: How was mitigation effectiveness verified?

In the end, despite some missteps and setbacks, we developed an advanced and highly effective application security program. Perhaps the biggest lesson we came away with is that developing an application security program is every bit as much about people as it is about the technology.

For further details on the development of my application security program, along with information on how most Veracode customers develop their program, please see my new guide, From Ad Hoc to Advanced Application Security: Your Path to a Mature AppSec Program.