The Secret Techie Blog

When I was hired to run the IT department of a major company my predecessor left three letters in the desk that was now mine. Each letter was clearly labelled; System Failure #1, System Failure #2, System Failure #3. A post-it note was attached to the bundle of letters.

In case of a substantial system failure open the letters in order, once per failure, and they will help you through the problem.

I put the letters back in the desk and forgot about them.

About one year later we had a cascading server failure that left our corporate intranet and several important production servers off-line. While repairing the problem I remembered the letters. Curious, I opened the first letter.

"Blame me, your predecessor"

The day after we got the servers back up I was called in to my boss's office to explain what happened and why were down for so long. Taking my queue from the letter I blamed my predecessor. My boss was satisfied with my answer and let me go.

About six months down the road we had another big failure. This time our primary database server went down and the secondary was having trouble dealing with the load. I had to put a lot of extra hours into getting them back up and we lost a few transactions due to the backup server not being able to function under the load.

Once again, I reached into that desk drawer and opened letter #2.

"Blame the equipment"

This time I lamented to the boss about how it wasn't my fault. It was that backup server! If we had some good equipment to run on these things just would not happen. He was satisfied with my answer and I went back to work.

Things ran smoothly for the next 18 months. Then we got hit with a virus that somehow got past our firewall and wrecked havoc on our systems.

I opened the third letter.

"Write three letters"

Original source: hmtk.com/blog/
Licensed under the Creative Commons Attribution-NonCommercial-
NoDerivs 2.5 License.