We have mentioned our favorite vulnerability scanning tools in this blog. But a lot of time has passed since, so it is time to put these tools against each other and evaluate the quality of the results received when scanning the same target.
UPDATE: After the constructive input from Michael A. in the comments, we have reworked the test for Nessus, to achieve more comparable results.


The Test Environment
The tested vulnerability scanning tools were installed on a Windows 7 Pro PC.

• Nessus server and client were installed and updated to the latest plugins.
• Retina 5.10.18.2135 Evaluation version was downloaded and installed. The Evaluation version does not allow updates, so we used what updates are included in the build.

The target was Damn Vulnerable Linux (DVL) version 1.5 installed as a VMWARE host with bridged networking on the same host PC as the vulnerability scanning tools. The network of the DVL target was bridged, and all firewalls (both of the host OS and the guest OS) were disabled. The DVL was started with the following services, with default settings and content as included in the distro.

• MySQL
• HTTP
• IPP Printer sharing which was active by default
  

The Scanning Process
Both scanners were started with setting on full port scan, with disabled safety of scanning, and all available plugins were activated. NOTE: Since Retina does not have WebApplication Analysis, Nessus was run twice, once with WebApplications disabled, and once with WebApplication enabled in order to do a meaningful performance comparison.
Performance

• The Nessus scanner without WebApplication scan took 8 minutes to complete the scan
• The Nessus scanner with WebApplication scan took 67 minutes to complete the scan
• The Retina scanner took 38 minutes to complete the scan

Results

• Both scanners failed to identify the target operating system
  

• The Nessus scanner identified the expected open ports, concluded that MySQL does not accept connections from unauthorized IP's. On a repeat scan, it regenerated the same results.
  
• You can download the full report of the Nessus Scan Here
  
• The Retina scanner identified HTTP and TCP port 631 (IPP Printer Sharing). It did not identify the MySQL port as open. On the Web server, it identified a significant number of vulnerabilites, but did not collect any information from the HTTP server. On a repeat scan it missed the HTTP port and only identified the MySQL port.
  
• You can download the full report of the Retina Scan Here
• The Nessus Scanner running the WebApplication Scanning repeated the previous results and additionally it identified a significant number of WebApp vulnerabilites, and collected information from HTTP through web mirroring.
• You can download the full report of the Nessus Scan with WebApplication Scanning Here

Conclusions
Both scanners performed a very well vulnerability identification but missed the OS identification. Also, both manifested flaws:

1. Nessus missed the IPP port every time
2. Retina manifested erroneous scan results, identifying different ports and vulnerabilities during different sessions - while no configuration changes were made to the test environment.

In terms of speed, without WebApplication Scan Nessus performed much faster then Retina. On the other hand, with active WebApplication Scan, Nessus was much slower then Retina.
In terms of scan depth, Nessus has a small advantage, since it includes a web mirroring tool that is very helpful in HTTP.

It can be clearly concluded that these tools cannot be used as the sole source of information when performing a vulnerability test. One must also utilize network mapping (NMAP, LanGuard), OS identification (NMAP) and specific application vulnerability scanners (ParosProxy, WebScarab for Web) for maximum effect.

In a direct comparison, Nessus wins because

1. Retina manifested erroneous results on repeat scans,
   
2. The Nessus package includes a WebApplication scanning module, which in eEye products needs to be purchased as a separate application

Talkback and comments are most welcome

Related posts
System Hardening Process Checklist
Web Site that is not Easy to hack - Part 2 HOWTO - the web site attacks
Checking web site security - the quick approach