It’s Just 1 Line of Code Change

Admit it, we’ve all been there. If you are shaking your head and saying nope, not me, you deserve a tip of the hat. The rest of us should have gotten the wag of the finger.  And we probably have already been punished with the disastrous results of our overly confident and/or overly optimistic action.

So, you’ve been working on implementing a particular feature, change or fix and you’ve gone back and forth coding, verifying, testing (I hope you didn’t skip the verifying and testing part). At this point, the code is so much ingrained in your mind that you know it forwards and backwards. It went through whichever formal testing process your organization/team has adopted. Everything looks fine and you are ready to release. At the very last second, somebody finds another, maybe very trivial issue or one of stakeholders asks you to just do a tiny change somewhere. The question is, “Oh how big of a change is it”, “Is it easy to fix?”  Since you’ve been living and breathing this piece of code for the past few days, or however long it took you to implement it, you reply “It’s just 1 line of code change.”  You are thinking to yourself, “Oh I’ll do the change and check it in, should be good to go.”  Right?  You are eager to release, stakeholders are eager to release, everybody’s eager to release.  Nobody wants to retest the whole thing for the one-liner, so you push to production…..

What’s the worst that can happen?  Well you might want to tell that to the NASA Engineers who worked on the Mariner probe launched on July 22, 1962. They might just punch you in the face or worse.

When the Mariner embarked on that fateful day, instead of making history in space exploration as the first rocket to fly by venus, it exploded and came crashing down after less than 5 minutes into the flight and went down in history as the most expensive programming typo.

Yes you read that right. All it was, was a TYPO. This typo cost the U.S. government about $80 million ($630 million in 2016 dollars). Somewhere within the computer instructions code a hyphen was omitted. So instead of soaring into space the Mariner ended up crashing into the ground.

Here is the official explanation provided on the  NASA website,

the Mariner 1 Post Flight Review Board determined that the omission of a hyphen in coded computer instructions in the data-editing program allowed transmission of incorrect guidance signals to the spacecraft. During the periods the airborne beacon was inoperative the omission of the hyphen in the data-editing program caused the computer to incorrectly accept the sweep frequency of the ground receiver as it sought the vehicle beacon signal and combined this data with the tracking data sent to the remaining guidance computation. This caused the computer to swing automatically into a series of unnecessary course corrections with erroneous steering commands which finally threw the spacecraft off course.

 

To summarize:

Use hyphen, have a successful space rocket launch, conquer venus, pop the champagne, have a parade, make history. 🚀  🎉 🍾

Omit hyphen, watch your rocket crash and burn, crush a nation’s dreams, waste the government’s money, live with the embarrassment till the end of your life.💥 😱 💸

Ok, so maybe the software you are working on might not be that mission critical. But that “omitted hyphen” equivalent in your one liner that you’ve just so carelessly released to production, might just cause you your very own mini Mariner disaster situation. Where in the best case you’ll have to scramble to fix the problem fast.

You may also argue that we don’t know if the “omitted hyphen” was caused by a last minute change. Probably not. It is still a great example though showing us how a tiny mistake can lead to a huge disaster.  So next time when faced with the question of doing that “1 line of code change” in the last minute, you just remember the Mariner and the hyphen. And be aware that releasing with the issue you know, rather than releasing with an unknown, might just be the best decision you’ve ever made.

We're Hiring Engineers at Ladders. Come join our team!