Backdoor Detection in the News

There has been some talk in the press lately about backdoors due to the recent court case where it was disclosed that federal agents planted a keystroke logger on a suspect’s computer using a trojan program. Many of the articles don’t report on the court case but raise the question as Declan McCullagh titles his article, “Will security firms detect police spyware?”

You can see the security cat and mouse game playing out between the police and suspected criminals although the roles here are reversed.The criminals are trying to secure their communications and the police are trying to break it to collect evidence.At first the police could just tap the data as it moved from sender to receiver. In response criminals started using strong end to end encryption.To get around that the police compromised the endpoint system with a backdoor to get at the clear text of the messages or passwords to log into encrypted servers.The criminal’s obvious response to that is to secure his system and try and detect any backdoor code.

Real detection of backdoor code is a challenging computer security problem to solve.This article will discuss different methods of detection and the different classes of backdoors.

Spyware, trojan keystroke loggers, or remote access trojans are what I call system backdoors, since they compromise the integrity of the whole computer system.These are typically in programs that you don’t want on your system. They are often installed via social engineering, a vulnerability, or a combination of both.

There is a different, more subtle and insidious backdoor which I call an application backdoor. This is backdoor code that is planted in an application that you do want on your system such as Wordpress, Borland Interbase, or tcpdump.All of those programs had versions that contained backdoors at one point. An application backdoor allows the attacker to bypass the designed authentication or authorization functions of the application and access its data and transactions.Sometimes an application backdoor is also a system backdoor if it allows functionality such as system commands and the application is running in an environment where the OS doesn’t prevent this.

For both system and application backdoors, once you know the program contains a backdoor you can make a signature for the program and detect instances of the program by computing a signature and looking it up in a signature database.This is how many traditional AV products find backdoor programs.The backdoor is typically detected by someone noticing a program that wasn’t supposed to be running on their computer.Then after inspection they also realize it is performing behavior on their system that they don’t want and report it to an AV vendor.

This approach doesn’t work well for custom or low population backdoors such as those used by sophisticated attackers or the police.It also doesn’t work well for application backdoors which are typically planted in the source code of the application or in the binary at the legitimate distribution site.Remember that applications backdoors are in programs that are supposed to be on the system so they don’t stand out.We need better was to detect backdoors than signatures.

Another way to detect backdoors is by looking for backdoor behavior on the system.Is a program listening on the network that isn’t supposed to?Is a program hooking into system level calls to gather keystrokes? Is a program hooking into system level calls to hide its behavior from the system log and administration tools.This is a big step up from signatures because it can detect a backdoor that hasn’t been seen before.

There are limitations to the behavior detection approach.If a backdoor program has done a good job of hiding its behavior from the system you can’t detect it.If a program is supposed to be listening on the network, detecting that doesn’t help.Some behavior may occur on a timer so the behavior detection would have to catch it in the act perhaps within very small intervals of time.Behavior doesn’t do a very good job at application backdoors at all.How will behavior find a special password that bypasses normal authentication or special processing when a certain client IP address is used?

A better solution to finding backdoors that overcomes many of the behavior approach’s limitations is binary static analysis. Binary static analysis can look at all of the functionality of the program without having to wait for it to get into a particular state which is the bane of dynamic (behavioral) analysis.

Binary static analysis is adept at seeing “baked in” static passwords, keys, or IP addresses. It can detect if there are encrypted or self-modifying blocks of code which may mean someone is hiding something. It can detect functionality such as listing on the network or shimming system calls.It can also do this from a trusted clean computer whereas the behavioral analysis needs to run on the same computer that the program being analyzed is executing.

Veracode’s binary analysis technology has scans that look for backdoors in executables.Chris Eng and I are giving a presentation titled, “Static Detection of Application Backdoors” at Black Hat in Las Vegas on August 2.We will cover detection of 4 classes of application backdoors that have:

1. Special credentials
2. Hidden functionality
3. Unintended network activity
4. Manipulation of security critical parameters

We will be giving a demo of proof of concept detection using IDA Pro and even screening a famous clip from a movie concerning backdoors.I’ll give you a hint, “Backdoors are not secrets!”