Go/No-Go time (when risk is involved).

At the end of last year I had an interesting experience with a project I was working on. Different people looking at the same situation would have very different opinions on how successfully the project handled this particular “situation”.

Without going into details, the objective of the project was to save the company money by modifying the infrastructure. A large number of changes need to be made to accomplish this, and if they were completed by a specific time a relatively large payment would not need to be made. If this deadline were to be missed, then the company(my client) would have to cough up the cash.

Timelines were aggressive, technical problems were discovered and needed to overcome, and while the business was all behind saving money, the business units who used the systems wouldn’t directly see the benefit, and so they were reluctant to allow work to be carried out. I suspect there was an attitude of “let the techies make their mistakes on other people systems, I want to go last”. Doesn’t work when most of the business is trying the same thing.

There was a point in the project where a decision had to be made to delay the project, or to go ahead and complete it. If the project were delayed, then the company as a whole would have to make this payment. If we proceeded, then people would physically remove hardware that was no longer needed (but only if the changes were done right.). Once the hardware was moved there was no going back.

A decision was due to be made by close of business Friday weather the project could complete. I was called on Wednesday afternoon and was asked to spend Thursday auditing the environment to verify that a certain function was working correctly on all systems. While on the phone I said I was pretty sure it wasn’t, and within half an hour I had quickly checked a system and replied via email that it definitely wasn’t. I also found a couple of emails I had sent over the last few months escalating a systematic configuration issue.

The long and the short of it was that there was a mistake in one of the early designs, and this mistake had been implemented repeatedly.

Thursday and Friday were long and busy days, but by the end of Friday we had a definitive list of all the systems impacted, and to what extent the impact was. Everyone knew that the cost of not going ahead was too large for them to delay the project, and the cynical side of me notes that some of the people making that decision had would have lost a lot of face. But the fact remains that when the decision was made to decommission the hardware the people making that decision know what the risks were, and how long it would take us to resolve the underlying issues.

While ideally this should have been addressed earlier, I am personally impressed that the customer made an informed decision. Of course, I don’t know how the business units would have felt had they been fully aware of the risks they were exposed to.