In a different life, I managed a team of professional services consultants. Part of my responsibility was recruiting these folks. For a while, I was moderately famous for my interview questionnaire, which featured stuff about megabytes this or tape speed that. I used this process to weed out resume packers. You know, the people who put something on their resumes if they'd ever seen it, leaned on it, or read about it in a magazine. If they managed to pass the test, the next step was an interview. After hundreds of these (yes, hundreds), I'd managed to tune the interview down to about a 20-minute process – especially essential when conducting 8 to 10 interviews a day.
So, how'd I do that, and why's that important? Backup administration was one of the big things we consulted on. As a professional services crew, we had to be at least as good as the best resource at a customer's facility. Otherwise, why would you hire us, right? Well, one thing I learned through these interviews is that asking an interviewee to do a self-assessment is possibly the best tool I had. I led with the question: "Rank yourself on your NetBackup [1] knowledge, 1 being novice, 10 being expert." [2] Most qualified respondents answered with something in the range of 5 to 8. Almost with no exception, the candidates who answered with a 9 or 10 were in reality nothing of the sort; neither expert nor qualified. All interviewees would be asked deep technical questions, but the initial depth was directly dependent on the professed level of expertise. Unsurprisingly, almost all "experts" failed within the first two questions. You see, I'd set a trap. From painful experience I know what it takes to be a really good backup administrator. I call it having the five fingers of administration. Really good backup administrators must have:
- Finger 1: A deep understanding of the applications in their environment. You can't back up Oracle, SQL, Exchange, and others without understanding how each one operates at the storage level. Often, you're required to be a qualified administrator/dba because the teams responsible for those applications don't know how to talk to the backup guys. Each application also had some rather esoteric and arcane dependencies with internal and external components. Not capturing those is a certain path to recovery disaster.
- Finger 2: In-depth knowledge of the operating systems you're interacting with. Almost nothing stresses your servers like backups. On top of that, each OS has peculiar things to worry about: how large directories are stored, how the system state is maintained, account permissions and privileges, etc. Each OS does things either slightly differently or in some cases very … ahh … uniquely. Unless you know how all this stuff works, you're guaranteed not to be able to do consistent recoveries. Even worse is our move to virtualizing everything. In this world, you could argue on what the OS is, and what state you're trying to preserve.
- Finger 3: In-depth knowledge of networking. I've personally never seen a better network diagnosis tool than backup applications. Nothing else consistently shoves as much data through the pipes. Oh, and if you want to restore the data quickly, almost nothing else is a bigger bottleneck. [3] Double oh, converged networks, clouds, and all the rest of the magic networking stuff make it even more complicated.
- Finger 4: In-depth knowledge of storage technologies. All the data you're backing up is sitting on some kind of fancy storage. It could be DAS, SAN, NAS, or even more esoteric stuff. Each have performance implications for backups and restores. Even worse, almost universally, storage vendors are integrating storage technologies with recovery. Naturally, the backup vendors are morphing their products to become managers or storage recovery technologies. [4] If you don't understand this stuff, you're most likely doing things wrong, inefficiently, and ineffectively.
- Finger 5: In-depth knowledge of backup and recovery technologies. If knowing the four other things wasn't hard enough, you also have to know the backup application you're using. This is no trivial task for any of the enterprise tools. Back in the day, I could teach a Networker or NetBackup class in five days. At the end of the class, the students would be credible admins. Today, it's not likely you could get the same level of effective knowledge in less than 3 or 4 weeks of intensive study. If you add practical lab work and real-world scenarios, this number at least doubles. This is true for all of the major products. The crazy thing is that during the time you spend learning this, the bar is moving and you have to learn more or even unlearn stuff almost constantly.
Knowing this made it pretty easy to stump the candidates who answered "9 or 10." They didn't get it: you'll never, ever be a 9 or 10. There's not enough time in the world to completelylearn each of the five fingers. The best you'll ever do is to get to know what you don't know and who to learn it from when you need it. By claiming a 9 or 10, the candidates opened themselves to me asking the hardest questions I could think of. That part was actually kind of fun.
As you might imagine, finding candidates with the right knowledge was really hard. But that wasn't enough. Knowing a lot of stuff didn't make them effective consultants. Knowing how to communicate was the hidden magic. It was a very rare candidate who was good at all of those. It also made those guys really expensive. [5]
And for those of you who are going to ask me, I rank myself about a 6 or 7.
[1] Or whatever backup product skill set I was looking to fill at the time.
[2] The astute reader will notice this wasn't a question. That was by design.
[3] Stay tuned for another post on this topic.
[4] The days of moving complete copies of data for every backup are quickly fading. Much is being done to keep you from having to move TBs and PBs of data. Stay tuned.
[5] Feel free to use this the next time you are up for a salary review.