Vulnerability Assessment tools differ greatly in their methodology for assessing a host. One might even consider the methodology as a sign of quality, but because most environments are fairly standard, the differences in assessment methodology are often invisible. The results produced by different tools are usually similar in the common cases because the most testing occurs with common cases. The common cases, however, don't exhibit the underlying methodology sufficiently.

Perhaps the best illustration of this principle, though it applies in many cases, is the much maligned 'HTTP banner check.' Let's say, for example, there's a vulnerability in Apache 2.0.55. Tool 'A' might test directly for the condition. Tool 'B' might check the version reported in the banner and report the vulnerability's presence if the banner matches. In this case, given no visibility into the functioning of the test itself, the results would be the same as long as the banner is accurate. The testing methods differ greatly in quality, but that difference is not necessarily visible. For a deeper understanding of vulnerability check precision, go back and read the post/whitepaper on the topic.

My point here is that if one is going to test assessment tools, one *should* test not only the common conditions, but also some of the uncommon conditions in order to surface differences in assessment methodology. Now, it should be said clearly that it is possible to fool any system. One simply has to determine what response a check is looking for, given a particular stimulus, and create a system that responds appropriately. IP360 is especially easy to fool in this manner because all of the rules used for testing are provided to the user. Other tools, which are not always open with their methods, require some effort to determine what exactly they're doing to test for a particular condition. Clearly, this sort of determined effort does not test the tools methodology, but only the tester's skills at interpreting the checks.

A more productive approach is to try some simple obfuscation techniques and see how the various tools respond.

Technique 1: No Banner

The concept of a product specific banner is commonly associated with HTTP. Most HTTP servers offer up a 'Server: ' item in the headers that usually provides application and version information. It's important, however, to recognize that the banner concept applies to other procotols as well. SMTP, FTP, and others provide very clear information in default banners. So, in testing assessment tools against null banners, try more than just HTTP.

When an assessment tool encounters, or its designers consider, an application with no banner information beyond what's required by the protocol, there are a couple of choices: 1. Proceed to determine what application is present without relying on banner information, 2. Give up and report the application as unknown, maybe through all applicable vulnerability tests for the protocol at it. In this case, clearly option 1 is preferred. A sophisticated tool should be able to detect the most common applications without relying on the banner being all chatty about what it is.

Technique 2: Inaccurate Banner

This is really an extension of technique 1, but an important one. Does the tool proceed intelligently when banner information isn't available? What about when the banner information is present, but isn't correct? There are a few fairly simple examples that apply here:

- Configure BIND to report an incorrect version in the version.bind string.

This can be managed with a simple change to named.conf (man named.conf). Using a very old version makes any erroneous results especially clear.

- Configure Postfix to return a valid Sendmail banner

Again, this is a simple change to a config file (main.cf).

- Configure Apache to return a valid IIS "Server: " string.

This requires source code modification and recompiling, but it's fairly easy to do. Configuring the Server header to return nothing is a simple change to httpd.conf.

An incorrect banner is more likely to result in false positives in the vulnerability results than a null banner.

Technique 3: Fun with CGIs

Testing for vulnerable CGI scripts can be difficult. Sometimes they're hard to get your hands on. The weakest test for vulnerable scripts is to issue a GET request for the file and check for a 200 OK response. So, try creating valid files with the name of typically vulnerable CGI scripts. For example:

- count.cgi
- jj.cgi
- loadpage.cgi
- sojourn.cgi
- webdist.cgi
- wwwboard.pl

There are many others. The key here is that they have to be valid CGIs, but just not the vulnerable ones. If a tool returns these as vulnerabilities, you can be fairly sure that it's not performing very rigorous checks.

Technique 4: The 200 OK Server

This is a variation of technique 3, I suppose. There are a fair number of HTTP servers that don't quite adhere to the RFC for HTTP. They will, for example, respond with a 200 OK to any request, regardless of its validity. So, a simple script that does the same provides a decent test. The better tools will figure out that the server exhibits this behavior and test it appropriately. The lesser tools will return a heap of HTTP false positives. For added fun, combine the 200 OK server with a false banner and see if any additional false positives surface.

These are just samples of some basic obfuscation techniques useful in evaluating vulnerability assessment tools. It's easy to go overboard with this stuff, but you quickly descend into the realm of breaking things just for the satisfaction of breaking them. There's nothing wrong with that, but it's always good to keep in mind the applicability of the testing to the actual project at hand.