It’s Just 1 Line of Code Change

Admit it, we’ve all been there. If you are shaking your head and saying nope, not me, you deserve a tip of the hat. The rest of us should have gotten the wag of the finger.  And we probably have already been punished with the disastrous results of our overly confident and/or overly optimistic action.

So, you’ve been working on implementing a particular feature, change or fix and you’ve gone back and forth coding, verifying, testing (I hope you didn’t skip the verifying and testing part). At this point, the code is so much ingrained in your mind that you know it forwards and backwards. It went through whichever formal testing process your organization/team has adopted. Everything looks fine and you are ready to release. At the very last second, somebody finds another, maybe very trivial issue or one of stakeholders asks you to just do a tiny change somewhere. The question is, “Oh how big of a change is it”, “Is it easy to fix?”  Since you’ve been living and breathing this piece of code for the past few days, or however long it took you to implement it, you reply “It’s just 1 line of code change.”  You are thinking to yourself, “Oh I’ll do the change and check it in, should be good to go.”  Right?  You are eager to release, stakeholders are eager to release, everybody’s eager to release.  Nobody wants to retest the whole thing for the one-liner, so you push to production…..

What’s the worst that can happen?  Well you might want to tell that to the NASA Engineers who worked on the Mariner probe launched on July 22, 1962. They might just punch you in the face or worse.

When the Mariner embarked on that fateful day, instead of making history in space exploration as the first rocket to fly by venus, it exploded and came crashing down after less than 5 minutes into the flight and went down in history as the most expensive programming typo.

Yes you read that right. All it was, was a TYPO. This typo cost the U.S. government about $80 million ($630 million in 2016 dollars). Somewhere within the computer instructions code a hyphen was omitted. So instead of soaring into space the Mariner ended up crashing into the ground.

Here is the official explanation provided on the  NASA website,

the Mariner 1 Post Flight Review Board determined that the omission of a hyphen in coded computer instructions in the data-editing program allowed transmission of incorrect guidance signals to the spacecraft. During the periods the airborne beacon was inoperative the omission of the hyphen in the data-editing program caused the computer to incorrectly accept the sweep frequency of the ground receiver as it sought the vehicle beacon signal and combined this data with the tracking data sent to the remaining guidance computation. This caused the computer to swing automatically into a series of unnecessary course corrections with erroneous steering commands which finally threw the spacecraft off course.

 

To summarize:

Use hyphen, have a successful space rocket launch, conquer venus, pop the champagne, have a parade, make history. 🚀  🎉 🍾

Omit hyphen, watch your rocket crash and burn, crush a nation’s dreams, waste the government’s money, live with the embarrassment till the end of your life.💥 😱 💸

Ok, so maybe the software you are working on might not be that mission critical. But that “omitted hyphen” equivalent in your one liner that you’ve just so carelessly released to production, might just cause you your very own mini Mariner disaster situation. Where in the best case you’ll have to scramble to fix the problem fast.

You may also argue that we don’t know if the “omitted hyphen” was caused by a last minute change. Probably not. It is still a great example though showing us how a tiny mistake can lead to a huge disaster.  So next time when faced with the question of doing that “1 line of code change” in the last minute, you just remember the Mariner and the hyphen. And be aware that releasing with the issue you know, rather than releasing with an unknown, might just be the best decision you’ve ever made.

We're Hiring Engineers at Ladders. Come join our team!

Why Do I Have to Test Your Shit? (The 5-Minute Sniff Test)

1*BJr8YvnbtRMw5idQ8pFCGA

If you can’t spend 5 minutes testing your work, then don’t make me.

I’ve spent 18 years in the software engineering industry, and have been called a programmer, then a developer, and now a software engineer — next is “delivery demon,” mark my words. In that time, I’ve tested my fair share of code. I’ve worked as a part-time tester, tested code I’ve written, ran QA groups, and poked my nose into systems I shouldn’t have.

There are many titles for those who professionally test software. You might be called a Tester, a Quality Assurance Analyst, or an Engineer in Test. You might be creating incredibly advanced automation scripts that produce beautiful reports via SonarQube, or you might be doing fully manual testing. Regardless of your title or approach, you are trying to discover issues before they make it to production, and users discover them.

A Tester is an Engineer’s partner in crime. Just like the editor of a book, you take what others (i.e. Engineers) have written and make it greater. As Steinbeck said of his editor Pat Covici, “He was much more than my friend. He was my editor. Only a writer can understand how a great editor is a father, mother, teacher, personal devil, and personal god. For 30 years Pat was my collaborator and my conscience. He demanded more than I had, and thereby caused me to be more than I should have been without him.”

Testers help Software Engineers go from Good to Great.

Over the years I’ve come to believe two things about testing:

First, what a tester does is hard. Their task requires the application of precision to a project’s specific requirements, unconventional thinking (box, what box?), channelling the end user, and a holistic/system-level view. Testers do an amazing job and have been critical to my success in releasing awesome software.

Second, a Tester’s job is made even harder by Software Engineers, because Engineers hand over shit code that doesn’t pass what I call the “5-Minute Sniff Test.”

The 5-Minute Sniff Test

As a Software Engineer, you chose a career building great software. You love solving fun and interesting problems using the latest frameworks, the coolest algorithms, and rainbows & unicorns. On a daily basis, “You Crush It!” However, all that screeches to a halt if you have to stumble through the muck of poorly defined (sometimes incomprehensible) requirements just to get to the heart of the problem.

Your progress is delayed, your time is wasted, and you’re pissed off! All of this could have been avoided if the Product Manager had just put a little more care into what they handed you. Doesn’t the PM take pride in their work? Do they even know what they are doing? Garbage in is garbage out.

When it is the Software Engineer’s turn to hand over their work, the code quality determines the efficiency, success, and morale of the person testing. The Tester expects the same level of quality as the Software Engineer expected from the PM. This doesn’t mean your code is bug free — we don’t want to be putting anyone out of a job — but it does mean your code meets a certain level of quality and professionalism.

How do we quantify the quality and professionalism expected? Enter the 5-Minute Sniff Test. Basically, if a someone (Tester, QA, Spouse, or Priest) can find major and obvious bugs in the first five minutes of testing your code, then you didn’t spend enough time testing. You’ve wasted your time, disrespected the Tester’s time, and started to build a reputation for delivering sloppy work (question: do you even know what you’re doing?). You laid a rotten egg that smells bad, everyone knows it is you, and you should be embarrassed. Garbage in is garbage out.

A Lesson from the Front

That “I want to crawl under a rock” feeling

My team and I once built an awesome new feature in our iPad app that was technically challenging and had the coolest design, and I was very proud of the work we did. It rocked and we had crushed it!

With a cheshire cat grin, I presented the app to our CTO of Asia (senior guy with about 10,000 people reporting to him). He started clicking everywhere except where I wanted him to click.

No. Don’t! STOP!

But he just kept clicking. The app crashed, data wouldn’t load, and mayhem ensued. He eventually handed the iPad back to me and grumbled, “This doesn’t look ready.”

He never actually saw the new features we had built. I was devastated, embarrassed, and wanted to crawl under a rock, but I had also learned an important lesson.

Take Care of Your Stuff

Don’t make someone else pick up after you.

Typically I’ll hear software engineers argue that it’s too hard to change their mindset from coding to testing.

That is why we have a testing team, right?

Bullshit!

I’m not asking you to walk, rub your tummy, and chew gum at the same time (it is harder than you think). I’m asking you to literally spend five minutes testing what you built and what you should be proudly standing behind.

Just because the city has street cleaners, doesn’t mean you should toss your candy wrapper onto the ground.

Don’t make someone else pick up after you.

We need to do better.

And we can.

How To Perform Your Own 5-Minute Sniff Test

The 5-Minute Sniff Test is the first line of defense against the “I will just hand over anything” mentality.

It means before you hand over your feature, stop, take a deep breath, clear your mind, and for the next 300 seconds, test what you just built.

  • Outside the Box — 90 Seconds: Did you step outside of your engineering shoes and test from a user’s perspective? Does the feature look nice and feel right, aside from the requirements? I’ve painted myself into corners arguing a feature is valid because it is “what the requirements asked for.”
  • Check Your Unit Tests — 30 Seconds: Do all the unit tests pass, and is there full code coverage? You can’t test everything by hand, so automated unit tests are vital for code quality. The 30 seconds allotted isn’t to create the unit test, but just to verify they pass. You should also consider failing the build if the unit tests don’t pass, which would allow more time for the obvious.
  • The Obvious — 190 Seconds: Did you test all the obvious cases? For example, if the pressed button should cause unicorns to dance across the screen, be sure to test if pressing the button actually makes unicorns dance across the screen. Do I get any errors in the logs, do rainbows show even though that isn’t part of the requirements (although it should be, because that would be awesome), and what happens if I double click the button? Review the requirements and make sure everything adds up. For example, did you catch that the total number of seconds listed here equals 310 instead of 300 seconds? Let’s correct “The Obvious” to 180 seconds.

Five minutes is all it takes to make sure you’re not handing over shit. If all else fails, grab a colleague and ask them to sniff around.

Looking Ahead

In his book “Blink,” Malcolm Gladwell relates the story of interviewers making snap judgements about the quality of a candidate within the first five seconds of meeting. The interviewer continues to subconsciously reinforce their initial impression by subjectively weighing the candidate’s answers towards their bias.

In other words, your first impression is your last impression. I don’t believe I ever fully recovered my reputation with the CTO of Asia after that bad first impression.

Performing the 5-Minute Sniff Test to check your shit takes very little time, but it can be of tremendous benefit to you, your colleagues, and your company:

  • It helps you build a Reputation for excellence and quality,
  • It Respects your partner in crime, the Tester, and
  • It is more Efficient with everyone’s time.

When everything looks and feels right after the 5-Minute Sniff Test, go ahead and hand off the feature you are proud of and can stand behind. You’ll have a thankful Tester, and have built yourself a reputation as an awesome engineer.

Originally published at Simple Programmer

We're Hiring Engineers at Ladders. Come join our team!

Visual State Testing with Mock Data

Testing is organized skepticism.

–James Bach

The Problem

Here at TheLadders we’ve been working on building new pages designed to provide job seekers access to data we’ve been able to collect over the years in new and interesting ways. These pages are static in design, but the amount of data available for each page can vary widely. This means we have a wide range of visual states for every element on these pages. Compounding the problem are occasional elements which impact the layout of their neighbors depending on their own state. All this brought us to a very clear need to test these visual states in an automated way.

Requirements

We wanted to cover all of our visual states. This meant we needed to make creating state combinations easy. Covering all of our states lets us add new elements to existing pages without worrying about breaking existing layouts. It also lets us modify shared styles and scripts without worrying about breaking layouts already using them.

We wanted to make sure these tests were automated and reproducible. They needed to integrate into our development, QA and release processes. Developers shouldn’t have to worry about remembering to run tests. We’ve written about how we accomplish this before with our friendly testing framework.

We wanted to minimize dependencies on services or data outside of our code. Tests shouldn’t force developers to jump through additional hoops in order for the tests to run. We can’t let internal network or DB issues stop development. And we can’t let code and data get out of sync for automated tests.


 

Our Solution

URL Parameters

To meet all of these requirements we use a set of optional URL parameters to force our server to generate mock data to exercise all of the UI elements on a given page. This lets us quickly define full page tests with different combinations of parameters to generate any and all possible states. These parameters are only respected in local and QA environments to keep our production site clean.

Example URLs:

For our companies pages, we expose the following parameters:

1
2
3
4
mockData : should the server return real or mock data?
detailLevel : controls the mock basic details data (description, location, details, etc)
salaryPoints : controls how many mock salary points to return
similarCompanies : controls how many mock similar company entries to return

Parameters generally limit their options to meaningful data densities. E.G. “full”, “minimal”, “empty”. This decouples the test code from the page layout details.

To force a layout where we have only limited data for a page, the company URL would look something like:

1
https://qa-1/companies/company-name/?mockData=true&detailLevel=sparse&salaryPoints=none&similarCompanies=minimal

And results in a page which looks like:

And to force a layout with lots of details and available data:

1
https://qa-1/companies/company-name/?mockData=true&detailLevel=verbose&salaryPoints=full&similarCompanies=extra

Resulting in a page like:

In order to test as much of the stack as we can and increase our test coverage for free, these parameters are passed all the way to the point in the code where we query our data stores, at which point the code simply has to check the flags to determine which data store, real or mock, to retrieve data from. This also means that the only code which has to change is the data retrieval code. No other server code and no client code has to worry about where the data is coming from. As long as it’s in the same format, they handle it just like real data from our data stores.

The Mock Data

Once we defined how we were going to request mock data, we had to decide how to provide it. The brute force approach of forcing developers to hand-code all desired variations wasn’t acceptable. This would have been a burden on developers and the end result would have been brittle. Any future updates to the data models would have forced developers to go back and update all of the previously defined mock data.

Our approach is to randomly generate all mock data. This way developers only have to define a generation function once per data point and then simply generate as many instances of those data points as are requested. This makes it much easier to cover all data states and eliminates hand coded data as long as you provide re-usable utility methods for string fields like names and titles.

The Code

Our server code is written entirely in Scala with Spring bindings.

Example entry point code:

Example entry point code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
  @RequestMapping(Array("/companies/{companyName}"))
  def companyPage(@PathVariable companyName: String,
                  @RequestParam(defaultValue = "false") useMockData: Boolean,
                  @RequestParam(defaultValue = "full") mockDetailLevel: String,
                  @RequestParam(defaultValue = "full") mockSalaryPoints: String,
                  @RequestParam(defaultValue = "full") mockSimilarCompanies: String ): Any = {

    val request = CompanyRequest(companyName,
                                 useMockData && Environment.current != Prod,
                                 mockDetailLevel,
                                 mockSalaryPoints,
                                 mockSimilarCompanies)

    val result = doCompanyQuery(request)

    buildView(result)
  }

Example data query code:

Example data query code
1
2
3
4
5
6
7
8
9
10
11
  def querySimilarCompanies(request: CompanyRequest): Seq[SimilarCompany] = {
      if (request.useMockData) MockData.similarCompanies(request.mockSimilarCompanies)
      else realSearchGateway.similarCompanies(request.companyName)
  }

  def similarCompanies(switch: String): Seq[SimilarCompany] = match switch {
    case "minimal"  => buildSimilarCompanies(5)
    case "full"     => buildSimilarCompanies(24)
    case "extra"    => buildSimilarCompanies(100)
    case _ | "none" => Seq.empty
  }

Example mock data generation code:

Example mock data generation code
1
2
3
4
5
6
7
8
9
10
  private final val BUILD_SIMILAR_SEED = 9284756

  def buildSimilarCompanies(count: Int): Seq[SimilarCompany] = {
    val rng = new Random(BUILD_SIMILAR_SEED)
    for (i <- 1 to count) yield {
      SimilarCompany(name = getRandomCompanyName(rng),
                     similarity = rng.nextFloat,
                     openJobs = rng.nextInt(50))
    }
  }


 

Gotchas

There are a few things to look out for with this approach. The first is to make sure the mock data generated is the same every time you run the test. This is as simple as seeding your random number generators and making sure to create them fresh for every request batch. If you don’t seed your random number generator you’ll get different results ever time, and if you don’t create a new seeded generator for each request batch the order of your tests will change what data is actually generated.

With all of this great UI test coverage it’s important not to neglect the code which actually queries the real data stores. Now that most tests never have to hit the data stores, it’s especially important to cover the data query code with their own unit and integration tests.

The flags and settings for mock data are specified at the entry points, but they’re not used until much further down the call chain. This can lead to mock-related code sprinkled throughout the code path if you’re not careful. In our current implementation we capture the mock flags in our entry points and store them in a single object which is passed to all data retrieval methods. This has been a nice simple approach for us which contains the flags to a single object. Each data-store gateway does still need to know how to provide mock or real data based on those flags, but keeping the mock data generation code centralized to one object per-page helps re-centralize the problem.

And finally we’ve found it’s best to have as many query switches as you have stateful elements on your page. This makes it easy to compose and maintain any number of tests from the individual mock data pieces. Most of our mock data generation is extended by our technical QA to help cover all of the UI states. It also lets us easily test components in isolation where necessary.

 


 

Conclusion

This approach to testing our visual states with mock data has proven invaluable as we build out more and more information-centric pages. I hope this approach finds a useful place in your arsenal of testing strategies on your current or future projects.