3 Rules to Prevent Backup Headaches

Any modern IT infrastructure needs and (usually) has a solution for backup of information. But due to the constant drive to reduce expenditures, very undesirable situations can occur, such as not being able to read the backed-up data.

Example scenario:
A telco company has two data centers- one operational and one warm backup datacenter which is kept in sync via replication services. Due to rise in capacity of stored data, the old tape library from the primary datacenter has to be replaced with a much larger tape library to accommodate proper backup.
The old tape library is still operational, and is moved to the warm backup datacenter to provide backup for the servers in the backup datacenter, should they become operational.
After 6 months, a major power failure occurs and the backup datacenter needs to be brought online. During the process, it is concluded that one of the ERP databases became corrupted during the replication and cannot be recovered. Since tape backup from the primary location is kept offsite in a bank vault, the tape backups of the ERP database are taken from the bank and brought to the backup datacenter. Upon attempting to restore the data from the backup tapes, it is concluded that the tapes are unreadable, and the database cannot be restored immediately.
The database is restored to an old backup and then rebuilt by manual entry over the course of 5 days.

Analysis:

  1. The vision of the backup systems operation for the primary and backup locations was that the servers at the respective location will backup to and restore only from the local tape library.
  2. Nobody bothered to check whether the old and new tape drives and tapes are compatible with each other and whether tapes from primary location can be read at the backup location and vice-versa.
  3. This led to the problem in which the last resort - the tape backup, although properly archived and protected was unusable.

Recommendations:
To avoid such and similar problems, follow these rules
  1. Make sure that you have full compatibility of all tape drives used within the organization - such compatibility will ensure that you can easily use any drive for any tape, even move one drive to a specific location if the need arises.
  2. Make sure that your tape drives are functional - perform regular 'exercises' of backup and restore of ALL drives within the organization. If you don't do this, by Murphy's law, the only remaining drive you have during an incident will be clogged up with dust or simply failed
  3. Make sure that your tapes are functional - perform regular 'restore exercises' for all tapes, and keep track of tape lifetime. The last thing you want is to have a possibly failing tape during a disaster recovery.

Talkback and comments are most welcome

No comments:

Designed by Posicionamiento Web