Testing of any solution, especially software is a very slow and painful process, which requires a lot of human resources and proper design of test scenarios. Because of the slowness of the process, something can be missed.
So a number of companies organize competitions in which they offer rewards to whomever breaches the security, finds a bug or similar activity to their software. Jon Oltsik in a text titled Carnival atmosphere in security compares this process to carnival and criticizes it heavily.
Although I agree that the competition is not a good approach, here is a more constructive analysis of the reasons:
Here are the perceived benefits of a competition style testing of any software. All of them are naturally legitimate, and every company would like an army of very dedicated testers for their product, at a price at which they'll never be able to hire so many testers:
- The application will get stress tested - An army of testers hunting the prize will put the software through it's paces and make very creative use cases to reveal bugs.
- A lot of boundary conditions will be checked and re-checked - These bug-hunting testers will throw all kinds of garbage at the software, precisely where most applications fail and open a way to fraud, security breaches or simply erroneous operation.
- Implementations of standard protocols and algorithms will be checked for errors - the most frequent path to breach of application or system security is the poor implementation of the standards that are trusted the most. For instance, while AES encryption is virtually unbreakable within an acceptable time frame, it's poor implementation in program code can lead to easy exploits and breaches. Such errors can be identified in a prize hunting test.
Naturally, it's not all good. There are several risks to using a competition for software testing, and here are the most very dangerous pitfalls:
- Only one winner - Only one vulnerability. This method of testing will actually identify only one vulnerability. There can be several controls to prevent this situation, like a submission time after which a winner is declared, but usually the first hacker to perform the breach will use covert channels to advertise his success, to deter other competition and to increase his reputation.
- Bugs publicized in the wild - A lot of other bugs and potential errors can be identified during the test, but these will not create the effect to win the prize. After the competition, information about these bugs can suddenly become available "in the wild" and the software company will not be aware of them. So a lot of people will get their hands on a list of bugs and can attempt some attack with them.
- A lot of potentially malicious groups can become very familiar with the software- In order for the competition to have any effect, the software needs to be distributed to everyone and their mother, no questions asked. This means that a lot of teams will learn the inner workings and the processes of operation of the application. In a standard test and delivery process such a corporate scale application, the general public does not know how to use it. While this security by obscurity is not too effective, it is another deterrent against attempts at fraud or attack.
There are visible benefits and a lot of good PR for anyone organizing a competition testing of it's solution. However, in the long run, the majority of valuable results of the tests will not be returned to the organizer of the competition, so there will be little benefit when the prize money is spent to close only one vulnerability.
So, unless you are up for a publicity stunt, when organizing a test, approach the process as a reward based test, but under controlled conditions (your testing facilities, scenarios to be filled and submitted for each bug found, etc.) and then reward the testers for each found bug.
Talkback and comments