Lessons Learned Building an Application Security Team

In 2012, I joined a large investment bank in London to start and grow its application security programme from the ground up. My initial focus was on the selection of the best tool for the job; namely, a static code analysis scanner that could be deployed easily, and scale widely. Within a few months, I had access to the Veracode Application Security Platform, and I was ready to start scanning my applications; however, I was missing one vital ingredient – my team. In the face of the security skills shortage, and so many struggling to manage programs with fewer people, I’m pleased to share my lessons learned (often the hard way) on creating this team. It was certainly a circuitous path, but in the end, the big, and surprising, takeaway was: develop soft skills internally, and supplement technical skills externally.

The Challenges in Building an Application Security Team

The challenge I faced was daunting: I had little organisational knowledge and was facing potentially hostile business units and development teams. In 2012, application security was a nascent practice; at the time, Gary McGraw described[1] the five ways to structure a “Software Security Group” based on BSIMM data as follows: software security services, security policy, business unit based, hybrid based or managed.

My initial view was to create a team that would be application security experts first and foremost and let the services and policy aspects of the programme simply fall into place around my AppSec team.

The Cyber Skills Shortage

Anyone involved in InfoSec is aware of the chronic shortage in skills; nowhere is this more severely felt than within the AppSec domain. A recent LinkedIn survey [2] found three openings for every four employed security professionals, whilst market demand for cybersecurity professionals has grown 90 percent from 2012 to 2015[3]. AppSec requires a unique (and uncommon) combination of both information security skills and application development skills, making this shortage even more chronic.

Building the Team

Starting With Software Developers

My standard reading at the time was Gary’s “Software Security: Building Security In” [4] in which he recommends to “start with software people.” Given I had a software background, this seemed an excellent approach. My initial job specification to my HR team specified “someone with 10 years software development in either C# or Java, an interest in security, hacking or reverse engineering” – I wanted to recruit a software developer with a security bent. I needed someone who could act in an advisory role, but have sufficient technical knowledge to ensure that my development teams would regard this person as a peer and not try and hoodwink him or her into bypassing our controls or security requirements.

Look for People Skills Internally, Technical Skills Externally

During the interview process, it became apparent that whilst I was attracting very smart and capable people, the job specification did not meet their aspirations; typically, they wanted to be far more “hands on” and actually get their hands dirty writing and fixing code – they certainly didn’t want to be wrestling with policy or compliance concerns.

Then Trying With Security Professionals

My HR team decided to widen the net and started bringing me traditional IT and InfoSec professionals. These were either from a risk or governance background (and not practical enough for my purposes) or from a network or operations background (the wrong types of skills for my purposes). Whilst such individuals are extremely technically capable, I knew they would not be suitable for my team – I expected to be spending the majority of time in conversation with developers, and I needed some who knew compilers, languages, etc.

How I Started My Team

Out of necessity, I recruited some individuals I regarded as a stop-gap measure. On the face of it, they had none of the requisite skills I thought I needed – having come fresh out of university, and in some cases, without any work experience. I was literally hiring a bunch of kids off the street!

Developing and Mentoring the Team

Once the programme commenced in earnest, it became apparent that the majority of the team’s effort revolved around being able to communicate with both developers and business owners, and far less about reviewing flaws and mitigation proposals. The soft skills, such as empathy, understanding and pragmatism, were far more valuable than out-and-out technical prowess. And this is where my “kids” really excelled: they were young and enthusiastic, very malleable in their thinking and willing to learn and develop. In a sense, they were very much undergoing the same learning process toward application security as many of our developers – there was a shared understanding and a willingness to cooperate and work together.

Running a highly technical programme such as static code assessment of an application estate is going to demand skilled technical resources. While our team only had a single resource (myself) with these skills, in the first two years, the team was able to very effectively utilise my engagement with developers. The team ensured that by the time I got to engage with developers, they were fully versed in the requirements and expectations – meaning that the meetings could be highly efficient. As the programme grew in scale, we were able to leverage the Veracode Application Security Consultants on an “as-needed” basis to step in and fill any peak demand for technical consultancy.

In my annual reviews with my team, one of the most common topics that came up was the fact that the team realised they were in a fast-paced and exciting environment affording them ample opportunity to learn and develop. The onus was on me to provide them with the appropriate training as needed. In turn, we ran frequent “Lunch and Learns” in-house (covering topics such as Application Development 101, How to Hack, How Does Veracode Work, Using Open Source Software, etc.), recorded webcasts of demonstrations for future learning, and I contributed to both internal and external wikis and FAQs on topics relating to the programme and AppSec in general. Frequently, we’d get the team to dial in as guests on consultation calls with security consultants so that they could get an appreciation for the types of questions a development team would ask.

The AppSec Machine

The numbers speak for themselves at the end of the day: this team was able to build a veritable “AppSec machine”[5] capable of scanning and remediating over 1,000 applications.

Whilst somewhat unorthodox in approach, my decision to build a team with a relatively low technical skill base and augment them with deep technical skills externally on an “as needed” basis meant that I could circumvent the traditional problems of cybersecurity recruitment (shortages, costs, and retention), and at the same time benefit from a team capable of high degrees of empathy and understanding. I would have no hesitation in doing the same again, or in hiring any one of this team again.

Read more about my experience building an AppSec programme in Ad Hoc to Advanced Application Security: YOur Path to a Mature AppSec Program.