Incrementally Improving Software Security Through AppSec Programs

The data on previously untested software makes a good case for regular testing and remediation of flaws. But initial scan data is only one element of assessing the risk readiness of an organization. Those in our sample set making the initial scan are taking only their first step in an ongoing process.

So what does that progress look like? The best one-word description is “incremental.”

Over the course of a year, we see that the overall body of applications saw a statistically significant improvement in pass rate at the latest scan as compared to the initial scan pass rate.

During the period covered by this report, the pass rate improved from 30.2% on initial scan to 34.1% in the latest scan (a 4 percentage point difference, but a 13% improvement rate). In other words, simply putting an application testing program in place can help an organization start chipping away at its flaws. This is further backed up by declining prevalence of the top 10 vulnerabilities in rescanned applications.

Some of the categories with the biggest decrease between initial scan and rescan were SQL injection (18% reduction in flaw incidence), encapsulation (17% reduction), and cross-site scripting (13% reduction). Not every category saw an improvement when we compared first scan to rescan – some saw small bumps upward in flaw incidence. There are a number of reasons for this – for example, the remediation process for other flaws could have potentially introduced other vulnerabilities elsewhere. However, across all of the top 10 vulnerability categories, our customers overall seem to have made progress by way of a 6% reduction in incidence of these vulnerability categories.

What matters most, however, is what kind of flaws make it into production, rather than how many of these flaws exist while the application is still in development and QA. This year we compared the incidence of the most common vulnerability types in both deployed applications and applications still in development. We did this by analyzing dynamic testing conducted on applications in a production environment versus those scanned in a pre-production environment behind a firewall.

The positive story here is that, for more than half the categories, organizations are making significant headway on reducing flaws in applications once they are deployed to production. The troubling element is that other significant categories go unchanged or even increase in prevalence in production. This could confirm the common complaint we hear from security teams that it is difficult for them to funnel requirements back into development once an application goes live. Frequently there's no active development team tasked with updating the application, meaning these flaws in production languish. Additionally, we've already noted that dynamic testing tends to dredge up different types of flaws. Dynamic testing also doesn't give developers the same code-level information that static testing provides, contributing to a longer remediation process.

The small overall gains in vulnerability prevalence in 2017 illustrate what veterans in the application security world have warned their bosses about for years now: it takes a while to get to OWASP compliance. This is no trivial effort. However, slow and steady progress can be maintained if organizations stick with it. Organizations that have been testing with Veracode since 2007 have a 35% better all-time OWASP Top 10 pass rate versus those who just started scanning this year (10 year-programs had a pass rate of 43% vs. 32% for first-year programs).

What Gets Fixed vs. What Really Matters

Overall pass rate improvement paints a basic picture of the state of software security. But ultimately, successful vulnerability management is all about prioritizing remediation based on risk. As such, we took a deep dive into the data that offers us a look at how well organizations are fixing the highest severity flaws.

The numbers are quite promising. When comparing the fix rate of very high and high severity vulnerabilities (37%) to the overall fix rate (19%), we see that organizations are reducing the most severe flaws at about twice the overall fix rate.

Slicing the numbers a different way, we find close to 77% of applications have at least one vulnerability (of any severity), on both initial scan and latest scan. It's pretty obvious what's going on with this lack of change: there's simply not enough bandwidth in IT to stamp out every single flaw within any given application. But rooting out high or very high vulnerabilities is definitely a worthy and often reachable goal, according to our statistics.

Comparing applications with at least one high or very high flaw at both initial scan and latest scan, we see that our customers made progress over the last year. The number of applications with high or very high severity vulnerabilities decreased by 26% between first and most recent scan.

Interestingly, though, the ratio of closed vs. open vulnerabilities in 2017 didn't track cleanly to the severity level of vulnerabilities.

In other words, we did not find that organizations closed a higher percentage of the most severe vulnerabilities. Instead, the highest percentage of vulnerabilities closed were very low, the next most closed vulnerability severity were high and medium flaws, followed then by very high flaws and then low severity flaws. Informational flaws, which do not affect the overall security of an application, but may indicate some underlying problem that requires more investigation, were closed at the lowest rate.

Similarly, when you look at the time it took to close flaws by severity, the highest vulnerabilities were not necessarily closed the most quickly. Very high severity flaws tended to take the longest to close, with 62% of closed very high severity flaws taking longer than 90 days to close; 72% took longer than 46 days to close; and 76% took longer than 30 days to close.

There are a number of explanations for this phenomenon. One is that many very high vulnerability flaws are also likely to be architectural flaws that are the thorniest to fix at the end of the development process or in production. Additionally, when we talk about prioritizing risk we must not only account for the severity of the flaw but the business value of the application at risk. Some organizations could also be potentially prioritizing a bigger chunk of high and medium flaws within higher value or more sensitive applications, compared to very high vulnerabilities in less sensitive applications.

Mitigation Reasons

Sometimes not all vulnerabilities closed are “fixed,” but are rather closed through mitigation. To understand why developers use mitigation to close flaws, rather than remediating flaws with a code fix, we looked at the logs for mitigation comments by developers.

Developers don't overwhelmingly reject static findings as false positives – they accept that the static analysis may be finding something. However, often developers have disagreements with static analysis based on different assumptions about how trustworthy their inputs are (e.g., databases and file stores). Other common points of friction include whether input from a third-party component may be trusted and whether functionality designed to be internal can be attacked.

The DevOps Effect

Folding security into the DevOps ethos of incrementally improving software in smaller, faster builds gets the AppSec world closer to its most cherished goal: fixing flaws earlier in the development process. Increasingly, our data shows a shift towards the adoption of DevSecOps practices.

Scan frequency is a good litmus test for DevSecOps adoption. Infrequent testing likely indicates an organization scanning once or twice as an assurance gate at the end of a Waterfall development process. Meanwhile, very frequent application scans are a sign that an organization is engaging in Agile or DevOps software delivery patterns, where security is tested and refined while the developer works on features and other functional requirements.

The data we examined for this edition of SOSS indicates that most of our customers are scanning applications more frequently. Overall, the data shows applications scanned 12 or more times per year (or, monthly on average) rose in prevalence.

In particular, the ratio of applications scanned only once or twice a year shrank the most noticeably. While the ratio of applications scanned quarterly remained relatively flat, the percentage of applications scanned on a weekly basis showed the biggest gains compared to last year (with a relative increase of 50% year-over-year). While this is a bit of speculation, this could potentially show a move by those scanning once or twice per year moving up to a quarterly cadence, while those scanning quarterly have shifted up to monthly, every other week, or weekly scanning.

Clearly, the percentage of applications only scanned a handful of times per year still makes up the bulk of applications tested (72% were scanned 1-6 times, and 37% were scanned just once in a year). The median number of scans across all of the applications inspected by Veracode tests didn't budge – it stood fast at two scans per year. However, the organizations that are scanning more frequently are increasing the velocity of their cadence dramatically, with the mean number of scans per application shooting up from 7.9 in last year’s report to 10.6 this year.

All of these gains in scan frequency are translating to measurable gains in the overall security of the software undergoing more regular scrutiny. There's no definitive set of numbers in our SOSS data that ties the iterative DevOps practices back to improved fix rates, but what we do have is very promising.

In addition to the on-the-record “policy scans” that are used to track compliance of software prior to release, the Veracode Platform offers what we call “sandbox” assessments. These sandbox scans aren't tied to compliance metrics and their output is private to the developer, who can scan their code as many times in the sandbox as they want. They're available so developers can test unfinished code early in the process without worrying about security staffers knocking on their door when early numbers don't meet compliance benchmarks.

What we've found is that when organizations take advantage of sandbox testing, scan frequency increases, and the reductions in flaw density are striking.

The difference between those organizations that use sandbox testing compared to those that only do policy scans translates to a 48.2% better fix rate when comparing flaw density changes from first scan to latest scan.

Watch: Tips for securing DevOps

Why is AppSec Still a Niche Discipline?

It's time to put the lazy developer trope to bed. It may be easy for cybersecurity pros to blame AppSec woes on indifferent, uncaring or slothful coders, but this hardly paints an accurate picture of what's really going on. And what's more, setting up this kind of adversarial dichotomy is hurting the security community's efforts to reach out to those closest to the development process.

The fact of the matter is that developers really do value security. According to the previously-mentioned survey we conducted with ESG, 62% of respondents said AppSec is very important to their development teams. Yet security may not be what developers’ bosses measure their performance on – just 18% in that same survey said security was the most important metric for measuring developers’ performance. What’s more, security education and ongoing training are not keeping up with the demand for secure coding skills. Veracode recently sponsored the 2017 DevSecOps Global Skills Survey from DevOps.com, and found that less than one in four developers or other IT pros were required to take a single college course on security. Meantime, once developers get on the job, employers aren't advancing their security training options, either. Approximately 68% of developers and IT pros say their organizations don't provide them adequate training in application security.

Despite the lack of resources for developers to boost their security skills, we’ve seen in our scanning data how additional developer resources can impact developers’ efficiency at fixing vulnerabilities. We’ve already discussed the effects of increased sandbox scanning on fix rates, but providing continuing education for developers on security topics also makes a statistically proven improvement in how well developers remediate flaws. When employers provide coders with opportunities to learn about application security through eLearning subscriptions, they stand to make a 19% improvement on fix rates.

Another best practice is investment in remediation consulting. One of the big challenges with security testing is that developers frequently misinterpret scan results or have a difficult time planning the next steps for fixing a flaw once the results are made available. Organizations that pick up consulting services that offer analysis and advice to developers alongside the scan results show tremendous improvement in fix rates. The numbers this year show that remediation consulting can contribute to a whopping 88% improvement on an organization's fix rate.

Clearly, if developers are given extra resources to accomplish their security goals, they will make progress on the flaw density in their software.

The big question left here is, what about integration of security tools with software development and delivery tools? The survey data we've collected in the past year has indicated that this stands as a stumbling block for improving the rate of security scanning in development shops. This year, we don't have the data to tie integration efforts back to remediation efficacy. But we're thinking about this metric and working on the instrumentation to measure it for future SOSS reports.

Watch: How to close the developer security skills gap

Ops People Impact AppSec, Too

It's also important to remember that developers are far from the only folks responsible for the overall vulnerability of an application to attacks. That's the whole point of the DevOps movement. We're all in this together in the struggle to improve software security, and that means that operations and other IT staffers play a big role in the process. That added dimension of responsibility has its own pros and cons. On one side, operations and IT staffers can help fix certain issues introduced by developers before applications go into production. But on the other side, ops people can introduce a host of other new problems into software.

For this year's SOSS report we took a look at the overall basic hygiene of the production environments on which applications run. What we found was that there were an alarming number of insecure servers running production software today. Even if these applications were flawless, they'd be vulnerable. We scraped this data by querying public-facing web applications on the companies' perimeters and logging what kind of messages servers responded with. These are the same kinds of responses that web applications would return to any browser or interested party requesting information from them – including malicious actors trolling for information about their targets.

One obvious best practice to prevent this kind of information leakage is to clean up server responses to hide all information about the server, or at least strip the server operating system version number from responses. Fortunately, about 37% of servers exhibited this kind of basic hygiene. The problem is that the next most prevalent response was from a web platform that's been around since 2009.

In fact, our examination found that 18.7% of sites were operating on web servers released a decade or more ago. And 25% of responding sites were running on web servers containing at least one vulnerability with a CVSS score of 6 or higher. Additionally, over a quarter of all sites responded with some sort of redirect, potentially indicating the use of shadow IT or sloppy IT practices.

In spite of these signs of sloppy deployments that are putting apps at risk, many operations people are also making a positive impact on software in production. As we discussed in the What Gets Fixed vs. What Really Matters section of the report, dynamic testing shows that apps running in production did slightly better than those in pre-production. Digging deeper into these numbers, we uncovered that the categories with the biggest difference in vulnerability prevalence between development/QA and production were those that were most likely under the control of IT ops. So issues like easily guessable passwords and the wrong use of HTTP security headers that can be tackled by ops are likely to be shut down before they go live.

Open Source Component Risk

Once again we examined the security of open source software components this year as a part of our SOSS analysis of Java applications. Though the results were slightly less dramatic than last year, they were nevertheless damning. In fact, we found that 88% of Java applications had at least one component-based vulnerability.

The crux of the issue appears to be that developers don't patch components in production once vulnerabilities are found and new versions of those components are released. In fact, according to our surveys over the past year, only 28% of organizations do any kind of regular composition analysis to understand which components are built into their applications. If more than two in three organizations don't even fully know which components they’re using, it's not hard to comprehend that so many of them remain unpatched.

In examining the most common Java components with high severity vulnerabilities (greater than a CVSS score of 6), we found quite a bit of turnover within the list from last year’s report. But it had less to do with patching and more with component feature upgrades, as well as newly found vulnerabilities that shook up the numbers.

“Struts-Shock” and the San Francisco Muni Attack

This is no theoretical problem anymore, either. Flaw-riddled components took center stage this past year as news broke of a number of high profile breaches caused by widespread vulnerabilities in open source or commercial components, including two big ones we analyzed. In March 2017, a number of high-profile targets got zapped by what we dubbed the "Struts-Shock" flaw. This critical vulnerability in the Apache Struts 2 library enables remote code execution (RCE) attacks using command injection, for which as many as 35 million sites were vulnerable. All it takes is a well-crafted web request to trigger the vulnerable code. The bad guys exploited the vulnerability in a range of victims’ applications, most notably the Canada Revenue Agency and the University of Delaware, in a breach of records that USA Today reported could cost the organization as much as $19 million.

According to our analysis, 68 percent of Java applications using the Struts 2 library were using a version vulnerable to Struts-Shock in the weeks following the initial attacks.

Just months before the Struts 2 zero-day attacks made waves, a ransomware attack against San Francisco Municipal Transportation Agency (SF Muni) likely targeting a deserialization flaw came into the limelight. This weakness was found in none other than Apache Commons Collections, a highly prolific component that is found seemingly everywhere in the Java world. Our figures show that approximately 53.3% of Java applications rely on Commons Collections versions (3.0 through 3.2.1 and 4.0) vulnerable to the deserialization flaw that left the SF Muni open to attack. Comparing the distribution of Commons Collections use this year to last year, we see more significant adoption of patched versions. And yet, a higher percentage of applications in 2017 were using vulnerable Commons Collections versions than in 2016, when 49% of applications had a vulnerable version.

The silver lining to these notorious attacks is that they're driving increased attention about the practice of composition analysis and the security of the software supply chain. Sentiment gathered from our survey work in the last year shows that 38% of organizations are evaluating the benefits of composition analysis as we speak.

Watch: Best practices to reduce risk from components

Identity Management

The mantra of “identity as the new perimeter” grows more relevant by the day, with the network perimeter irrevocably dissolved and applications growing more connected than ever. CISOs increasingly look toward identity-centric security as a top concern right alongside application security. Security teams are investing in centralized identity and access management (IAM) to tie together the threads of access control across hybrid cloud environments, and to feed into more advanced security monitoring like user and entity behavior analytics (UEBA). But these risk management tools won’t be effective if the applications themselves have glaring flaws that put the assurance of access control in question. These worthwhile IAM initiatives could suffer from a very vulnerable AppSec Achilles heel.

The unfortunate fact of the matter is that, as IAM has grown in prominence among security leaders, so too has the prevalence of flaws that put identity controls at risk, particularly credentials management. Also keep in mind that this is data based on automated scanning. There are a limited number of authentication and authorization problems that can be found reliably via automation, so this represents just a subset of the potential risk in these areas.

Comparing Industry Performance

This year we carved up industry breakdowns slightly differently, with the changes most notable in two categories, infrastructure and government. Infrastructure is a new category grouping that we created based on organizations in energy, transportation, and utilities. Given the increased focus by the security community on improving the posture of assets in the critical infrastructure technology stack, we felt it was important to start tracking the state of software security in this grouping. Meanwhile, we massaged the government category to include federal, state, and local governments, as well as education organizations, under a single umbrella. As with the overall OWASP pass rate, industry pass rates upon initial scan are down considerably across the board in 2017.

Most industry groups underperformed compared to the overall OWASP pass rate benchmark this year, although the newly organized infrastructure category, manufacturing, and healthcare came within a couple percentage points of the benchmark. Those looking for silver linings will also note that the lowest performing industries in last year’s SOSS study, government and healthcare, experienced the smallest declines in pass rates year-over-year. That lining may be the merest of slivers for government, when you start looking at incidence of some of the major vulnerability categories on first scan.

The numbers for vulnerability prevalence on first scan shows that government was in worst place in nearly every category. The only exception was cryptographic issues, for which healthcare showed up in last place. On the other hand, the technology sector showed amazing performance for these major vulnerability types. This chart offers a glimpse primarily at vulnerabilities that are still in the development process, so there is a ray of hope here given how much less prevalent these vulnerabilities are than in the overall data set, which includes production applications.

When we add latest scan pass rates into the mix, we do see a modest increase in applications passing policy. Interestingly, while infrastructure performed best on first scan, and was in the middle of the road with latest scan numbers, its pass rate numbers remained essentially flat from first to latest scan. This could potentially indicate problems in infrastructure organizations’ ability to remediate found vulnerabilities, which mirrors industry hearsay about the challenges around improving the posture of legacy software like SCADA systems. As this was our first year analyzing infrastructure industry data, we’ll be sure to monitor this critical area in future reports to pick up on trends.

This year we also took a peek at how many applications within an industry were undergoing their first policy scan as compared to the rest of the portfolio under current testing. A higher percentage of new applications undergoing their first policy scan tends to suggest that those organizations are getting started in the application security maturity process.

Similarly, a lower number of applications getting their first scan could indicate that customers in this grouping have already covered a lot of their portfolio and are further along on the maturity curve. However, it could also indicate that the industry is struggling to remediate the applications it already has under testing and doesn't want to add any further tests until it can prioritize fixes in its existing portfolio.

For example, though it is historically one of the most mature sectors in application security, financial services is pretty low on the list. But it is not the absolute lowest when it comes to on-boarding applications – government is. Given government's annual worst place position in OWASP pass rates in our SOSS reports, it is probably understandable why these organizations are hesitant to expand their portfolio under test.

Meanwhile, healthcare and retail/hospitality on-boarded the most applications relative to the size of their portfolios. This could go a long way toward explaining their good performance in remediation from first scan to latest scan. With so many new applications added, these industries likely were able to take care of a lot of low-hanging fruit, namely easy-to-fix flaws that were newly found. Next up, we’ll take a deeper dive at some key statistics in each of the major industry categories.