Warning: That backup server I configured failed silently for 6 months

I set up a secondary backup server for a medical office in Phoenix back in March. Thought I had it solid with daily syncs and alerts. Turns out the sync job stopped after 2 weeks because of a permissions change nobody told me about. No email alerts, no logs in the dashboard. I only caught it last week when I needed to restore a file for a client and got nothing. Lost a full days work because I had to rebuild from the primary server logs. Now I manually check my backup servers once a month with a test restore. Has anyone else had a silent failure like this bite them when they least expected it?

3 comments

3 Comments

ross.kim7d ago

Lost a full days work" - man, I feel that pain. I read somewhere that something like 80 percent of backup failures are silent, meaning no alerts go off and you don't know until it's too late. I had a similar thing with a tape drive backup once where the drive just stopped writing after a firmware update but still reported every job as successful. That's why I started doing random spot checks with real file restores on different dates, not just checking the logs or the dashboard. It's the ONLY way to be sure.

ross.kim7d ago

Wait... the drive just kept saying everything was fine while actually doing nothing? That's terrifying, I never even thought to check if a firmware update could break things that badly. Makes me want to go double check all my backup schedules right now.

troyc177d ago

My buddy had the same nasty surprise, @ross.kim. Found out the hard way after a crash.