By now, you probably know that details of the DNS vulnerability have leaked. Halvar Flake speculated on DailyDave and the momentum built from there, despite the fact that his guess was short on a few key details. I don’t need to rehash the full technical details here; by now, they are easy enough to find with a couple Google searches. When Slashdot picks up the story, it’s hardly a secret any more.

What’s more interesting to me, now that I’ve digested the big secret, is how this whole situation has played out in the security community.

The security community has been polarized for the past two weeks, not so much over the technical details being withheld, but about Dan’s plea that people not speculate about the vulnerability. As many pointed out, the “bad guys” won’t stop trying to figure it out just because the “good guys” keep quiet. To be honest, my own lack of public speculation wasn’t because I agreed with the philosophy; I just wasn’t smart enough to figure out the vulnerability myself.

People implied — or stated outright — that Dan just didn’t want anyone stealing his thunder. Considering the timing of the release and the subsequent BlackHat talk, it’s obvious why such accusations were made. Personally, I think it’s a little of each. I believe the coordinated patch effort was undertaken with the best of intentions, but I also think Dan relished some of the glory and media attention as well. It’s hard to blame him for that; if you were in his shoes, wouldn’t you want some recognition too?

By many accounts, dealing with the DNS vulnerability from the operational side has been an exercise in frustration. Plenty of IT people wanted to patch but couldn’t get approval without being able to justify the operational risk. “Because Dan said so” is apparently not a convincing enough argument. Some wondered why the people who were responsible for creating the problem should be blindly trusted to implement an appropriate fix?

Ultimately, vulnerability disclosure is a minefield. No matter how you choose to disclose, somebody will always disagree.

P.S. If you didn’t figure out the title of the post by now, Nate was one of the unlucky few to draw the same timeslot at BlackHat as Dan Kaminsky.

A co-worker passed along this snapshot taken at the Karsten Nohl, Jake Appelbaum, and Dino Dai Zovi talk at HOPE this past weekend. The context, of course, is that the overzealous Debian developer who accidentally crippled OpenSSL back in 2006 said he did so because valgrind reported uninitialized memory use. Click through for the full-size version.

So automated software review is dangerous now? Perhaps that bullet should read “modifying code you don’t understand is dangerous.”

Despite what various commenters around the blogosphere think (I’ve read a few but can’t find the links now), Dan Kaminsky’s online “Check My Dns” utility doesn’t:

• Poison anybody’s DNS cache
• Expose how the actual exploit works

What it does is check whether your ISP’s DNS server is patched. Plain and simple. It looks for one thing — source port randomization. This does not give away the exploit, it checks for the existence of the sledgehammer fix that prevents the exploit from working.

More specifically, there’s some Javascript code that generates a random hex string which is used to create a URL, e.g. http://6313d97e498e.toorrr.com. Your OS then does a DNS lookup for that unique hostname. Your ISP’s DNS server asks toorrr.com’s DNS server (a server Dan controls) to resolve that funky DNS name to an IP address. It sends a few packets in the process. Dan’s server makes a note of the source port of each request and sends back the webserver’s IP address to your DNS server, which sends it back to you.

Now that you have the IP address, your browser can fetch the results page. The web page is generated dynamically by parsing the hex string out of the URL you requested, using Ajax to fetch the relevant port and TXID data stored on Dan’s server, and printing out a “safe” or “vulnerable” message such as:

Your name server, at 71.243.0.38, appears to be safe.
Requests seen for 6313d97e498e.toorrr.com:

71.243.0.38:45298 TXID=13926
71.243.0.38:45310 TXID=25412
71.243.0.38:45338 TXID=30829
71.243.0.38:45332 TXID=13934
71.243.0.38:45321 TXID=2701

That’s all. Nothing tricky. This particular DNS server is deemed safe because the source port varies from one request to the next.

Come to think of it, those source ports don’t really look that random, do they? For anybody “in the know”, is that amount of randomness sufficient to protect against the attack?

The security community is cynical. So much so, that most of the chatter that’s taken place over the past 24-36 hours has suggested that Kaminsky’s DNS vulnerability was little more than a publicity stunt and that his BlackHat presentation would be an over-hyped rehash of prior art. Granted, one has to suspend disbelief to even consider that something monumental would be discovered in DNS — that’s the protocol itself — but hell, it’s always nice to give a guy the benefit of the doubt.

Faced with nearly a month of criticism and questioning, and understanding the persuasive power of a technical peer review, Dan decided to expand the inner circle, so to speak. Rich Mogull arranged a phone call with Tom Ptacek and Dino Dai Zovi so that Dan could spill the beans and let them decide for themselves whether it was spin or substance. Turns out there was substance.

Now we sit around and wait until August 6th to cram into a ballroom with a thousand sweaty conference-goers to hear the juicy details. And Dan’s presentations are usually packed to the brim even when he’s not announcing anything.

In the meantime… how about patching those servers?

Rich Mogull’s executive overview of Dan Kaminsky’s latest DNS vulnerability fluffed a few feathers yesterday:

The good news is that due to the nature of this problem, it is extremely difficult to determine the vulnerability merely by analyzing the patches; a common technique malicious individuals use to figure out security weaknesses.

The typical response I heard was “what do you mean, it can’t be reverse engineered? I’ll just look at the diffs!”

In hindsight, after examining the BIND diffs (yes, I did it too) and discussing with colleagues, all most people saw was UDP source port randomization and a better PRNG for generating the transaction ID, the latter of which would appear to be related to Amit Klein’s cache poisoning attack from about a year ago.

What Rich was really saying is that you can reverse engineer the patch until you’re blue in the face, but that won’t reveal the specifics of the vulnerability.

Dan’s blog post this morning appeared to confirm that interpretation:

DJB was right. All those years ago, Dan J. Bernstein was right: Source Port Randomization should be standard on every name server in production use.

There is a fantastic quote that guides a lot of the work I do: Luck is the residue of design. Dan Bernstein is a notably lucky programmer, and that’s no accident. The professor lives and breathes systems engineering in a way that my hackish code aspires to one day experience. DJB got “lucky” here — he ended up defending himself against an attack he almost certainly never encountered.

Such is the mark of excellent design. Excellent design protects you against things you don’t have any information about. And so we are deploying this excellent design to provide no information.

To translate the fix strategy into a more familiar domain, imagine large chunks of Windows RPC went from Anonymous to Authenticated User only, or even all the way to Admin Only. Or wait, just remember Windows XPSP2 :) This is a sledgehammer, by design. It cuts off attack surface, without necessarily saying why. Astonishingly subtle bugs can be easily hidden, or even rendered irrelevant, by a suitably blunt fix.

Nate McFeters appears to think that Tom Ptacek has figured it out. I’m going to go out on a limb and say that Tom didn’t figure anything out yet but still wanted to write a pithy blog post. I think that if Tom had figured it out, he would have written it down privately and posted the SHA-1 hash, as is the trendy thing to do these days.

Speculation aside, the title of Tom’s blog entry, Dan Kaminsky could have made hundreds of thousands of dollars with this DNS flaw!, does make an important point — Dan didn’t sell the details to ZDI, he used his influence and reputation to coordinate a massive vendor patch effort. That’s an admirable move.

I’m finally getting around to finishing my post on minimizing attack surfaces. Here’s Part 1, in case you missed it.

First, a quick clarification. I noticed that some of the readers who commented on that first post wanted to talk about improving security through the use of various development methodologies or coding frameworks. Those are interesting tangents (and ones that I may write about in the future), but my intention with this post is to discuss a very specific problem related to how people integrate third-party code — that is, the stuff you import or link in but didn’t write yourself.

As I mentioned previously, developers have a tendency to “bolt on” third-party components to applications without understanding the security implications. Often, these components are glossed over or ignored completely during threat modeling discussions. I attempted to illustrate this with my fictitious WhizBang library example in Part 1.

When integrating a third-party component, developers familiarize themselves with the API but generally don’t care how it’s implemented. Granted, that’s how an API is supposed to work; you don’t have to futz around with code beyond the API boundary, and you can blissfully ignore parts of the library that you don’t need. In past consulting gigs, I’ve sat in threat modeling discussions where nobody knew whether a particular library generated network traffic. “We just use the API,” they say. The fact that it works is good enough; nobody seems to care how it works.

That mindset is ideal for rapid development but problematic for security. Failing to understand the complete application, as opposed to just the part you wrote, prevents you from accurately assessing its security posture.

It’s also no coincidence that web app pen testers love third-party components — we get excited when we see “bolted on” interfaces, because we know that developers tend to leave extraneous functionality exposed. The resulting findings usually generate reactions such as “I didn’t even know that servlet had an upload function.”

An Example

Here’s a close-to-home example related to my post about DWR 2.0.5 from the other day. DWR is an Ajax framework that has a variety of operating modes. In-house, we use a subset of DWR’s full functionality — specifically, we interact with it using the “plaincall” method only, so we made sure that the features we didn’t need were disabled via the configuration file. As it turned out, there were vulnerable code paths prior to the “do you have this thing disabled” check. In hindsight, if we had taken more time to understand the exposed interfaces, we could have reduced the attack surface by filtering out unneeded request patterns before they even touched the third-party code.

But wait, you say. What about maintainability? If I whitelist using a point-in-time application profile, doesn’t this create the same maintenance headache as the reviled WAF? It doesn’t have to. Certainly, one option would be to whitelist each and every unique URL that references the DWR framework, e.g.

/dwr/call/plaincall/myMethod1
/dwr/call/plaincall/myMethod2
/dwr/call/plaincall/myMethod3

But then you’d have to update the whitelist every time you added or removed functionality from your application. Also, don’t lose sight of the security goal, which is to minimize the amount of exposed third-party code. If I add or remove URLs that list, provided they are still using the “plaincall” method, I’m hitting the same DWR dispatcher every time. So I’ve increased maintenance cost without any security benefit.

A better option is to simply tighten the URL pattern a bit in the J2EE container. Here’s the default configuration:

<servlet-mapping>
  <servlet-name>dwr-invoker</servlet-name>
  <url-pattern>/dwr/*</url-pattern>
</servlet-mapping>

Now, instead of allowing every URL starting with /dwr/ to be processed by the DWR library, you could be a little more restrictive:

<servlet-mapping>
  <servlet-name>dwr-invoker</servlet-name>
  <url-pattern>/dwr/call/plaincall/*</url-pattern>
</servlet-mapping>

In this configuration, you don’t have to worry about /dwr/call/someothercodepath any more. There is less third-party code exposed, thereby reducing the overall attack surface of the application. (NB: DWR also serves up a couple of Javascript files, so those URL patterns will have to be whitelisted too)

A Logical Extension

Even if you’re not a developer, you should still be thinking about attack surfaces. People download and install blogging platforms such as WordPress, Movable Type, etc. all the time, but how many take additional steps to harden their installations? The concept is the same as the OS hardening analogy I brought up at the very beginning of this discussion.

Similarly, people install third-party WordPress plugins or Joomla components without considering that most of them are written by some random programmer who is a whiz with the plugin API but knows nothing about security?

At the risk of sounding trite, always remember that security is only as strong as the weakest link.

Popular Mechanics recently published an article about the NSA Red Team, which caught my interest, having been a part of that organization for a short stint back in early 2000. The article does a decent job of describing the Red Team’s charter, which is essentially to attack DOD targets in an attempt to simulate real adversaries, not unlike a consultant running a pen test against a corporation. The rules of engagement are similar to most pen tests: don’t DoS the target, don’t install malware, generally be non-destructive.

Disappointingly, the author sprinkles the usual super-secret uber-hacker spin throughout the article to make the Red Team seem mysterious and exclusive, with untouchable talent. It’s a little misleading. For starters, there’s the predictable question about success rates:

I’d heard from one of the Department of Defense clients who had previously worked with the NSA red team that OWNSAVAOG and his team had a success rate of close to 100 percent. “We don’t keep statistics on that,” OWNSAVAOG insisted when I pressed him on an internal measuring stick.

This is one of those statements that is difficult for the average reader to interpret. It’s intended to make the team sound like a crack squad of hackers, but in reality it’s the same statistic that every security consultancy cites during sales calls. The truth is, there’s a lot of wiggle room on what is considered “getting in” to the target. For example, some would say that brute forcing an FTP server and downloading some FOUO (For Official Use Only) documents constitutes penetrating the target. Others would disagree.

How about personnel? I thought this was an englightening and accurate statement from the unnamed NSA source:

And like any good geek at a desk talking to a guy with a really cool job, I wondered just where the NSA finds the members of its superhacker squad. “The bulk is military personnel, civilian government employees and a small cadre of contractors,” OWNSAVAOG says. The military guys mainly conduct the ops (the actual breaking and entering stuff), while the civilians and contractors mainly write code to support their endeavors. For those of you looking for a gig in the ultrasecret world of red teaming, this top hacker says the ideal profile is someone with “technical skills, an adversarial mind-set, perseverance and imagination.”

He basically admits that the team consists mostly of people who “run the tools” and only a handful that actually write the tools or do anything cutting-edge. It shouldn’t be that surprising; just as in any large consulting organization, you have some people who run scanners/tools and aren’t expected to be terribly analytical. While the Red Team almost certainly has some superstars, on the whole it is similar in both skillset and composition to a typical consultancy or enterprise security team.

In terms of attracting and retaining top talent, the Red Team faces the same challenges as the rest of the information security industry, with the built-in disadvantage of the government pay scale. If that wasn’t bad enough, they also have to compete with themselves (i.e. the rest of the NSA) for already scarce resources. Given these challenges, how could one realistically expect the Red Team to be as advanced as the article portrays?

Finally, let’s dispel the “super-secret” notion — unless things have changed significantly, the majority of Red Team operations are unclassified. Granted, detailed information is guarded, but you can find reports summarizing past operations if you dig around a bit. One would expect that an operation intended to be truly secretive would never make its way into Google search results.

I want to conclude by saying that this post is not intended to cast the Red Team itself in a negative light. I enjoyed my time there and had the opportunity to work with some smart people. The Red Team’s goals are worthy and noble; clearly, state-sponsored cyberterrorism is a growing concern and as a country we should be as prepared as possible. But realize that we have a long way to go.

We all know it happens, but it is rarely exposed as clearly as Adam Pennenberg did in his article for Fast Company, The Black Market Code Industry. It turns out that this 0day seller was an HP employee:

According to the consultant who snared Marester, his quarry’s skills appear quite sophisticated. His wares, if they performed as advertised, could help a hacker take down machines running that particular software anywhere in the world. His real name is Steve Rigano; he’s a self-employed network consultant from Grenoble, France, who works full time at HP, where he is listed in the switchboard and maintains an hp.com email address. He told me that he saw nothing wrong with offering tools and techniques that targeted the company providing his paycheck.

A self-taught hacker, Rigano says he discovered the vulnerabilities and coded the exploits on his own time, which he says is none of HP’s business. “I have the right to sell what I want,” he says. He told me he attracted mostly Chinese and Russian buyers, but claimed he never found takers for the HP or SAP “vulns” and exploits. He said he stopped selling black-market code in January but didn’t explain why.

Most security companies I have been acquainted with frown on this type of activity, as I am sure HP has. It’s hard for them to sell security products and services when their employees are selling the very tools the company is purportedly defending against.

[Update 7/7/2008: The information in the Fast Company article is being disputed by Steve Rigano. He has notified us that he has taken legal action against Adam Pennenberg and Fast Company.]

DWR 2.0.5 addresses an XSS vulnerability that is likely to be exploitable in most 2.0.4 installations. If your web application uses DWR’s Ajax implementation, download and install this update now!

As an aside, I’ve been a fan of DWR for a while now, not only because of its ease of integration but also because it was the first Ajax framework to offer built-in CSRF protection. You could tell that Joe Walker was taking security seriously. For this particular vulnerability, I e-mailed him on a Saturday night, and within 12 hours, he had confirmed the problem, patched the code, and built a 2.0.5 release candidate. Granted, it was a tiny code change, but I’ve still never seen a response that fast. Less than a week later, the official 2.0.5 release was tested and available for download.

That’s it for now, but I’ll be referencing this example again when I get around to writing Part 2 of my Minimizing the Attack Surface post.

This post is a response to Alan Shimel’s Topic of Interest #2 for the Security Bloggers Network.

So what motivates me to attend BlackHat? The #1 reason for me is networking — meeting new people and catching up with old friends and colleagues. Despite our best intentions, we are all busy and our networks are constantly expanding, making it increasingly difficult to stay in touch with old friends in the industry. Twitter and other forms of microblogging help you chip away at the communication gaps; you get a glimpse into peoples’ lives but it’s no replacement for a real conversation.

Obviously, the briefings themselves are a major draw. Even though it’s expanded to over 10 tracks now, the quality hasn’t suffered much. This year’s experiment with allowing paid delegates to vote on speakers seems to have produced a good lineup, though I’m sure there was still a selection committee that could and probably did overrule the votes in some cases. Either way, BlackHat presentations are a decent indicator of the overarching themes that will be prevalent in information security for the upcoming year or two.

When I first started attending BlackHat, I was drawn to the talks discussing 0-day vulnerabilities, tool releases, shellcode tricks, and the like. These days, anything relating to static analysis, automation, and of course web security are most interesting to me. I also consider who’s speaking, regardless of the topic (e.g. one of these guys presents, I’m there). In general, I’ll try to gauge how much value the speaker will add to the presentation — in other words, what do I gain by attending the talk vs. flipping through the slides later? I never attend every time slot; sometimes the hallway conversation is just more interesting.

Some of my other reasons for attending, in no particular order, most of which fall under the “networking” umbrella:

• The parties (duh)
• The Pwnie Awards
• Meeting fellow security bloggers
• Recruiting speakers for SOURCE
• Finding future Veracode employees
• Trading war stories
• Picking up vendor schwag for my kids (RSA is much better for this one)
• Meeting current and former customers — and future ones, hopefully

Things I could do without:

• The cigarette smoke
• The heat
• Quark’s

I’ve stuck around for DEFCON a couple times in the past, but I don’t anymore. I fly out Friday morning or early afternoon so I get home in time to spend the weekend with the family. Personally, three days in Vegas is plenty for me.

When it gets closer to BlackHat time, I’ll post my picks from the briefings schedule.