Live from AWS re:Inforce: Learnings from Security Enablement for DevOps at AT&T

https://www.veracode.com/security/web-application-security-testingThis week, AWS ran its inaugural security conference AWS re:Inforce in Boston. There were several interesting talks at the conference, and I found John Maski’s presentation, “Integrating AppSec in your DevSecOps on AWS,” contained great practical advice. Maski worked for AT&T for 32 years, with his most current role being Director, Production Resiliency & DevSecOps Enablement. He recently joined Veracode to advise customers on how to best integrate Veracode into their security pipeline, and we’re lucky to have him on the team.

Support from Executive Leadership is Crucial

Starting out, and as expected for any large organization, Maski found a huge variety of skill levels and a lot of variation in how people ran their development pipelines outside of the central DevOps initiative.  Software development was optimized for speed – aka “quantity” – and security was an afterthought. 

On the upside, Maski saw pockets of advanced knowledge and CI/CD implementations. A significant CI/CD platform was already in the works. Most importantly, there was a huge appetite among executives for making quick and extensive progress.

“In an organization the size of AT&T, you can’t make meaningful progress without the support of executive leadership,” Maski said. “It is absolutely critical to drive the necessary cultural changes.”

With this backing, he set out to connect with partner organizations, working collectively towards the seemingly impossible goal to secure AT&T’s entire application landscape. Spoiler alert: When Maski recently left AT&T, they were very close to completing this goal.

Integrating Security into the CI/CD Pipeline

If you are coming from the security side of the house and are in charge of application security, it really pays off to truly understand your organization’s development tools and how pipelines are set up. Not only will you be able to speak your engineering team’s language, you will be better suited to advise them on how to integrate security testing solutions.

Most of application security can and should be automated, with the exception of what’s at the very beginning and the very end of the process. Threat modeling is still a manual process that relies on human understanding of the architecture, even if there are tools that help visualize and document this process. Likewise, penetration testing is a final litmus test at the end of the development process that should be carried out on any critical application before it is deployed into production.

In the middle are various automated testing solutions that should be run automatically to regularly provide feedback on security defects. Static analysis tests the application code for a broad range of security flaws, and it can be fully automated into both the IDE and the CI process. In the IDE, it provides early security guidance and education to software engineers while they are coding by highlighting potential vulnerabilities and suggesting best practices. Veracode has found that integrating SAST in this early stage in the process has helped organizations to reduce newly introduced flaws by 60 percent.

However, guidance at this stage is not mandatory and is mostly suitable to removing flaws in newly written code. To ensure a more structured feedback and compliance process, static analysis should be integrated into the SDLC. Typically, development teams would scan as part of their CI process, either on a code commit or a pull request, and get security defects flagged through the ticketing system. They will do this scan in a “sandbox,” so that results do not get escalated to the security team. Finally, for high security applications, we recommend doing a scan on the full scope of the application before each deployment to ensure that no security defects escape to production.

Software composition analysis looks at known vulnerabilities in open source libraries that are being used in the code. If you find such a vulnerability, the fix is usually upgrading to a different library version rather than fixing the open source code yourself. SCA often integrates with the SDLC in the same places as static analysis.

Dynamic analysis is a third way of looking for vulnerabilities in software and is typically applied to web applications. Unlike static analysis, which looks at the application code, dynamic analysis interacts with the application via an instrumented browser that crawls and audits the application. While findings with other testing solutions overlap, there are several security issues that only dynamic analysis can detect, including server configuration errors. Dynamic analysis is typically run in the QA stage against a staging server and against the production server.

Five Tips for Getting Traction with Your DevSecOps Initiative

With many lessons learned having during his DevSecOps initiative at AT&T, Maski shared his five recommendations to get traction with your own program:

1. Partner with stakeholders: Identify, collaborate, and align with your partners, especially in software development. You have to understand their world and respect their point of view for your program to be successful.
2. Pick the right metrics: Know your metrics before you jumpstart the program. Talk upfront with your sponsors and partners on what success means to them and agree on metrics.
3. Don’t boil the ocean: Go “Agile.” Pick pilot applications to secure, so that you can learn from the process and expand to the next group of applications. Keep note of what you learn along the way to improve the program over time.
4. Run an internal campaign: Communicate effectively to raise awareness about the importance of AppSec to the business. Tie AppSec to the mission of the company. Use your communication to educate DevOps team members about AppSec to help strengthen their expertise.
5. Demonstrate progress: To ensure continued executive support for your program, regularly report your program’s progress through the metrics you picked. Tailor your progress reports to the audience; for example, your senior leadership will want to see different metrics than your engineering partners.

Key Learnings from AT&T’s DevSecOps Program

Maski left the audience with three key learnings from running his program:

• Strong Executive Leadership is key to driving the necessary cultural changes – and to secure the required budget.
• If getting your program started quickly is a requirement, use services built on a robust platform. That way you can focus on onboarding applications rather than building and maintaining scanning infrastructure.
• Build a strong team and have a flexible plan. Map out and communicate your plan with confidence. That doesn’t mean that your plan has to be perfect – learn and adjust as you go along. Set bold goals to drive progress.

Veracode was and is a cornerstone of AT&T’s AppSec strategy. If you’d like to learn how to build an AppSec program in your organization, download The Ultimate Guide to Getting Started With Application Security.