I have over thirty years of experience in IT Systems Administration, and yet, the other day I made a mistake while updating a server. When the updates went south on me, I reverted to an older snapshot instead of the one I took before starting the updates. Months worth of data on that system were instantly gone.

I didn’t even realize my mistake until a few days later when a user asked if those system updates could have deleted data. I immediately knew what I had done wrong.

Now, most of our systems are backed up on a nightly basis, but this was a development server which was excluded from the backup routine because it took up too much space. When I realized that this server wasn’t available in backups, I started to panic a little bit. Without a backup, there was very little chance that I would be able to recover the data. Maybe someone made a copy of the data and saved it to another system, but that would be a longshot.

At this point, I wasn’t worried about getting in trouble for this mistake; things happen. However, I felt a knot in my stomach for letting down our users in such an irresponsible manner. I should know better than that. I would feel awful for weeks.

Thankfully, mostly by luck, the storage system had a protection plan that included nightly snapshots. I mention luck because I intentionally skip the protection plan with other development systems. I mean, why back it up if it’s just a test server?

Since the volume had a snapshot from the evening before I messed everything up, I was able to recover all of the missing data. It only took an hour from realizing my mistake to restoring the data, but I did sweat it a bit.

There are two lessons I take away from this experience that transform it from a negative into a positive. The first and probably most important point is to always, and I mean ALWAYS, back up your data. It doesn’t matter if it’s just a development or a test server. If people are using it, then it’s important enough to protect the data.

The second lesson is that even though I have decades of experience, I am still prone to making simple mistakes. And this was a reminder that I still need to be careful when I’m working on systems.