Quality feedback loops

You can get the best results from your quality program by putting in some effort to keeping your feedback loops tight.


Consider this example, of a software application or firmware release. Unit tests, which are typically run by the developers on their functions after any changes, are superb. The time delay is nearly zero, and the developer can investigate any failures nearly immediately, and often with the help of the unit test framework and any debugging tools that are part of their toolchain.

A short feedback loop has the huge benefit that the problem is still fresh in the developer's brain and they don't have to re-learn things that they were doing days or weeks or months ago!

Automated tests are great, and are a huge growth area. These can holistically test the whole application or firmware package, and simulate user interactions. They are often run when each build is finished, unless the previous run is still going on. Sometimes they're run daily, kicked off by hand, or at other intervals.

Manual testing is still very valuable, especially when it comes to audio or video quality or any kind of subjective test. Automation can detect problems with a test, but a human tester will also very often catch problems that aren't specifically related to a test they're performing, for example a screen-drawing problem that the automation wasn't programmed to watch for.

Many larger software projects have a beta program, where software is released early to a handpicked set of users, who have typically signed an NDA or confidentiality agreement. Selecting users based on the quality of their past bug reports, or their value as a customer is common. As you can see in the diagram, though, it can be challenging to fix problems reported at this late stage.

Finally you get to your end users: the longest feedback loop of all.

It happens. You did all the testing you could, but after the product is released, the complaints start ramping up. You missed a serious problem. A "hot fix" is often implemented to get your users a working version without having to wait for the next release. Starting with the build that was released, limit your fixes strictly to just the one big bug. Resist putting in anything else! They will just expand the risk and the amount of time you need to test!

Now, update the other tests in the loop so that next time you'll catch that problem way earlier!

Comments

  1. UnknownJune 5, 2019 at 2:07 PM

    I find one danger with automated tests is the often-incorrect perception that they are really testing the whole problem space. Some functionality lends itself well to automated testing (e.g., the feature "does not crash"), but other behavior is too complex to meaningfully test in a fully automated way. This can lead to "looking where the light is better" and an overemphasis on the easily-automated tests, leaving it up to your Beta testers and end users to find those difficult problems (which also tend to be the more difficult ones to analyze and fix).

    ReplyDelete
  2. Chris OwensJune 9, 2019 at 8:15 PM

    Agree; it's very easy to imagine test automation as a fortified wall that no bugs can get through; but automation has profound weaknesses that can be very difficult to notice before they bite! The military has a concept of "defense in depth," where even after a force gets through the perimeter, there are defensive systems that can take care of them. A great way to think about software testing, I think. The automation is the perimeter, but you're going to need some other testing as well.

    ReplyDelete

Post a Comment

Popular posts from this blog

The Boeing 737 MAX, a QA perspective

Image
Thanks to: Wikipedia.org user Acefitt The Boeing 737 MAX and its MCAS system (which tries to emulate the handling characteristics of earlier generations of the product) have been in the news a lot lately. Investigations are still ongoing, but the product is being blamed by many for two crashes which have resulted in the loss of almost 340 lives. That's a big quality problem. Clearly, the pressures on the Boeing are immense. The worldwide commercial aircraft market is over $191 billion , and dominated by cut-throat competition between Boeing and Airbus. The 737 has been in production since 1967, and if you are a pilot who was qualified to fly it in 1967, you still are today , despite vast numbers of changes to the aircraft. This is because Boeing has managed to keep the aircraft handling similar enough over time that the FAA (and other worldwide governmental authorities) has been willing to allow the "type certification" of pilots to be valid for all variants of t...
Read more

The risks of the "Golden" sample

Image
It's very tempting, and if you've ever worked with overseas contract manufacturers (CMs) they will almost prefer to use a "Golden Sample" instead of a detailed specification. Coyau /  Wikimedia Commons  /  CC BY-SA 3.0 In hardware manufacturing, it's easier to understand. I freely admit I've written incoming parts inspection instructions that don't have every detail of the part! Every dimension, the depth of any indentations, the gauge of any round protuberances, the color of all the materials, the hardness, even the chemical composition (important for fasteners and structural parts). If you simply take a prototype or early unit from the manufacturing line, give it a thorough inspection, and declare it to be a "Golden Sample," it's a heck of a shortcut, saving a lot of *your* time as the person who would otherwise have to write the documentation! There are risks, however, and you need to think about them, minimize them, and try to m...
Read more

Applicability of Economics to QA

Image
Years ago, I started listening to Russ Roberts' excellent podcast EconTalk ( http://www.econtalk.org ), and the more I've listened to it, the more I've come to understand that although the origins of economics and the study of quality may be different, the practice is very, very similar. What do we measure? We face a constant struggle trying to measure the right variables, and what is "right" changes over time. In economics, it seems like new indicators become popular all the time. After the Great Depression, the concept of a nation's Gross Domestic Product rose in popularity as economists worked on trying to explain how nations were affected by the depression, and how they might be able to avoid similar events in the future. However, GDP did nothing to predict the "Great Recession" of 2007-2009, and in fact trailed the crisis by a considerable time. We face similar issues with quality. Production or build test failure rates come to us p...
Read more