Don't Just Assume Your Backup is Working!

Many small businesses have invested in a backup solution to protect their critical business data in the event of accidental deletion or system failure. There will come a day in your business when such an event happens.

At my work we lost our system administrator about a year ago and hired a local computer service company to ensure that the transition went well and that our company interested where taken care of. As part of their evaluation we where sold a backup server because our existing one couldn't manage the size of backup required to get a full backup daily. They organized people inside our business to switch out the tapes daily and take them home on alternating days of the week. Neither of these people have any IT experience. They where told eject the tape and put the tape in for that day. Late one Thursday afternoon one of these two people came to me and said, "I guess out backups have been failing for about a week. The IT service company has isolated our issue to the fact that we have outgrown the capacity of our current tapes." Immediately we began planning to purchase more tapes to fill the void. They arrived on Monday and I thought they where placed into the rotation to fill the gap so we could get a full backup. We got a partial backup on Monday night the part that would fit on one tape. The second half was scheduled to backup on Tuesday night.

Tuesday morning my boss came to me and said that a spreadsheet that she'd been working on for the past 3 years is gone. She didn't know when it was deleted but thought she had it late the week before. I called the IT company thinking that we should have at least one full backup remaining somewhere out there. I asked for the copy of the file from the night before. They quickly contacted me back stating that the requested file wasn't backed up on Monday night it was scheduled to be backed up on Tuesday. So I opened up my request to go further back. Please give me any copy of this file from the most recent full backup available I asked. Once again I received word back that no copy of this file is available on any backup. Now I have to tell my boss that a spreadsheet that contains information she had been compiling for 3 years is unrecoverable. This immediately raised an alarm for us. What data is backed up if we can't recover a simple spreadsheet that was on a file share? Is our database at risk? How about our e-mail server? If something more serious happens right now we are likely completely unprotected.

I tell you all of this so you might reconsider how protected your data is and think about ways to avoid this type of issue from happening to you.

We found that we had several points of failure. No one ever tested our backup solution since it was purchased over 1 year ago. We only had 5 tapes total in our backup rotation. The last full successful backup was overwritten due to lack of instructions given to the people who managed the tape exchange. We relied on the service of an external IT company without a written contract detailing what we expected of them including our backup protection, recovery, and testing. The last point of failure was the fact that no one who knew that the problem was happening for a week alerted people higher in the company that we had a potential issue until it was to late.