Nessus vs Retina - Vulnerability Scanning Tools Evaluation

We have mentioned our favorite vulnerability scanning tools in this blog. But a lot of time has passed since, so it is time to put these tools against each other and evaluate the quality of the results received when scanning the same target.
UPDATE: After the constructive input from Michael A. in the comments, we have reworked the test for Nessus, to achieve more comparable results.


The Test Environment
The tested vulnerability scanning tools were installed on a Windows 7 Pro PC.

  • Nessus server and client were installed and updated to the latest plugins.
  • Retina 5.10.18.2135 Evaluation version was downloaded and installed. The Evaluation version does not allow updates, so we used what updates are included in the build.

The target was Damn Vulnerable Linux (DVL) version 1.5 installed as a VMWARE host with bridged networking on the same host PC as the vulnerability scanning tools. The network of the DVL target was bridged, and all firewalls (both of the host OS and the guest OS) were disabled. The DVL was started with the following services, with default settings and content as included in the distro.
  • MySQL
  • HTTP
  • IPP Printer sharing which was active by default

The Scanning Process
Both scanners were started with setting on full port scan, with disabled safety of scanning, and all available plugins were activated. NOTE: Since Retina does not have WebApplication Analysis, Nessus was run twice, once with WebApplications disabled, and once with WebApplication enabled in order to do a meaningful performance comparison.
Performance
  • The Nessus scanner without WebApplication scan took 8 minutes to complete the scan
  • The Nessus scanner with WebApplication scan took 67 minutes to complete the scan
  • The Retina scanner took 38 minutes to complete the scan
Results
  • Both scanners failed to identify the target operating system

Conclusions

Both scanners performed a very well vulnerability identification but missed the OS identification. Also, both manifested flaws:
  1. Nessus missed the IPP port every time
  2. Retina manifested erroneous scan results, identifying different ports and vulnerabilities during different sessions - while no configuration changes were made to the test environment.
In terms of speed, without WebApplication Scan Nessus performed much faster then Retina. On the other hand, with active WebApplication Scan, Nessus was much slower then Retina.
In terms of scan depth, Nessus has a small advantage, since it includes a web mirroring tool that is very helpful in HTTP.

It can be clearly concluded that these tools cannot be used as the sole source of information when performing a vulnerability test. One must also utilize network mapping (NMAP, LanGuard), OS identification (NMAP) and specific application vulnerability scanners (ParosProxy, WebScarab for Web) for maximum effect.

In a direct comparison, Nessus wins because
  1. Retina manifested erroneous results on repeat scans,
  2. The Nessus package includes a WebApplication scanning module, which in eEye products needs to be purchased as a separate application


Talkback and comments are most welcome

Related posts
System Hardening Process Checklist
Web Site that is not Easy to hack - Part 2 HOWTO - the web site attacks
Checking web site security - the quick approach

6 comments:

M.A. said...

Nessus web app tests are enabled. They are known to be very slow, so comparing the scan speed does not make sense.

Many Nessus scripts failed to run (they display general information at the end). The Nessus test was obviously interrupted or your installation is broken. Comparing the content of the reports is irrelevant too.

I cannot comment the Retina report, I do not know this tool.

Bozidar Spirovski said...

Reset - I'll run the scan with a fresh install of Nessus

M.A. said...

You still have a problem with Nessus. This is not normal:
WARNING : no port scanner was enabled during the scan. This may
lead to incomplete results
Scan duration : unknown (ping_host.nasl not launched?)


Did you enable at least one portscanner? Otherwise, this explain why some results are missing.
You should also make sure that your plugin set is up to date (run nessus-update-plugins if necessary)

Anonymous said...

I am sorry, but you tesing is flawed. eEye does not support Retina on Windows 7 with the version you have listed in your article. You should revisit your testing procedures and requirements.

Bozidar Spirovski said...

That was the latest version of retina available at the moment of the test. My approach will remain the same - scanning the same target with the latest available tools, but using win7 will become a rule, not an exception.
I do plan to revisit the scan, and extend it to some more tools for good measure.

scottjames said...

good android phone or tablet can run. Bluestacks has actually Google Play Store for PC Download Windows 8.1/10/7 Free should certainly root their device. Numerous Android tools nice.

Designed by Posicionamiento Web