All large businesses rely on software solutions to manage many aspects of their business. In time, the business grows so accustomed to the "system" that they deem it authoritative for all information and rarely question it's outputs. When this point of trust is reached, any error in that system can go unnoticed and have a tremendous impact on the organization, even put it out of business.
A Billing system, Provisioning system and a General Ledger is the core system of an electrical supply company IT support systems. The billing system calculates the bills according to usage data records, the provisioning system activates and deactivates customers and the payments of bills are processed through the General Ledger System system, and are matched to the information in the billing system.
Due to marketing strategy, the company defines a change in billing model for one of their products, to include the ability of no subscription charges under certain conditions. The manufacturer of the billing system dispatches consultants to configure the new product into the billing system.
The consultants configure the new product, and reconfigure the billing parameters for the product. The product is tested successfully for proper billing and rolled out. It is a commercial success, and significant number of customers are using the new product.
A month later, the Board of Directors requests a report on the number of customers who haven't paid their bills . The Provisioning system is generating these reports from specific markers in the customers account within the billing system. The generated report indicates large number of customers of the modified product have not paid their bills.
Based on this information, the Board publishes an advertisement warning all non-paying customers that they will be disconnected. After 2 weeks, the report still shows no change in number of paid bills, so it is decided to disconnect these users for a warning period of 2 hours.
The following morning the disconnection is executed. By the end of the day, at least 400 customers have filed lawsuits against the company for breach of contract, since they have paid their bills.
The ensuing internal investigation identified the following problems:
- When modifying the product, the consultants actually created a second version of the product which didn't have subscription charges.
- This new product was integrated into the GL system for payment processing, but due to oversight, the process didn't mark the bill as paid in the billing system.
- Nobody bothered to check whether this marker is being updated, assuming that everything is automatic.
- The provisioning system looked at the erroneous markers showing unpaid bills, but nobody doubted the produced report, nor made any effort to make manual or different automatic verification.
Conclusions and Recommendations:
As stated in the introduction, the real cause of this incident is the unquestionable trust in the system. In a broader aspect, the following mistakes occurred:
- Poor implementation by the consultants - they implemented a change and didn't properly communicate the performed change
- Poor testing and verification by the utility company - the testing scenarios did not include all relevant aspects, like the paid bill marker for provisioning
- Blindly trusting an obviously unnatural report by both operational teams as well as the board - nobody who had access to the report actually identified a problem with it, although it is very hard to comprehend that SUDDENLY NOBODY PAYS his bills.
- When implementing system changes, obtain verification and advice from all interested parties - in integrated systems, everyone uses data from everyone else. So technology architects or system custodians should confirm that the change affects their system in a expected manner.
- When testing system changes, include standard tests from all interested parties - like the previous point, everyone should fire a battery of tests from their own point of view. This is the best way to identify integration risks
- For all critical and business impacting reports and functions, implement an automatic comparison control that observes the information from a different aspect - for the example above, another report from GL listing the number of payments to the utility company could have been compared to the report produced by Provisioning. An immediate discrepancy would have been identified
- Institute regular operational on-site controls with verification of a sample set of system information to paperwork - where automatic systems aren't available, perform regular manual verification with a sample set of data
Talkback and comments are most welcome