Startup Quality

You can get the best results from your quality program by putting in some effort to keeping your feedback loops tight. Consider this example, of a software application or firmware release. Unit tests, which are typically run by the developers on their functions after any changes, are superb. The time delay is nearly zero, and the developer can investigate any failures nearly immediately, and often with the help of the unit test framework and any debugging tools that are part of their toolchain. A short feedback loop has the huge benefit that the problem is still fresh in the developer's brain and they don't have to re-learn things that they were doing days or weeks or months ago! Automated tests are great, and are a huge growth area. These can holistically test the whole application or firmware package, and simulate user interactions. They are often run when each build is finished, unless the previous run is still going on. Sometimes they're run daily, kick...

{This is a post I wrote for the public tech blog at one of the companies I have worked for, OnLive, a cloud gaming service. It's aged pretty well, and the company is no longer around to stop me, so I'm publishing it here!} Software QA is full of pragmatic choices and best guesses. However, the ubiquity of pass/fail tests doesn't help at all. How can we make automated test results clearer, more reliable, and more useful? In many companies, you can find two main areas in which testing takes place: Unit testing of modules and executables at check-in­ time, sometimes as part of a continuous integration process and  Integration and system testing, performed on a test instance, or controlled subset of your whole site or system.  Both categories can benefit from being more mindful about the data behind the pass/fail result. Let’s take a few example cases: a) Verify a new user can complete the sign­up process. Although a frequently-changing UI might make this a good ca...

It doesn't happen too frequently. A lot of times it's a slow-build of customer complaints. The product is unstable. It keeps crashing. It slows down. It's freaking out. But you just can't reproduce it. How can you get traction on an intractable bug? Museums Victoria You've got to start somewhere, and try to find a reproducible problem. Maybe it's not the whole problem, but some part of it. Review the reports, maybe even contact the original users and try to focus your efforts on specific places in the UI or the workflow. You don't need to find the whole "unstable" problem. For example, let's imagine that the UI displays an unexpected screen. Just reproduce that. Just reproduce it in one place. You can start at screen X, under certain conditions, and try to go to screen Y, and end up somewhere else. If you have to write 1000 words of "here's how you get set up to reproduce this issue" then so be it, but find one thing you ...

Years ago, I started listening to Russ Roberts' excellent podcast EconTalk ( http://www.econtalk.org ), and the more I've listened to it, the more I've come to understand that although the origins of economics and the study of quality may be different, the practice is very, very similar. What do we measure? We face a constant struggle trying to measure the right variables, and what is "right" changes over time. In economics, it seems like new indicators become popular all the time. After the Great Depression, the concept of a nation's Gross Domestic Product rose in popularity as economists worked on trying to explain how nations were affected by the depression, and how they might be able to avoid similar events in the future. However, GDP did nothing to predict the "Great Recession" of 2007-2009, and in fact trailed the crisis by a considerable time. We face similar issues with quality. Production or build test failure rates come to us p...

Pete Birkinshaw Having a flexible library of documents is a wonderful thing. Having a full corporate document-control system with email notifications and approvals gives you a certain amount of power, but I feel like the "sweet spot" for tech and startups is just storing documents in version control, such as Git or Subversion (svn). These systems are often well-backed up and available remotely. In addition, they carefully track changes, can display differences between versions, and you can easily "revert" to an older version of the document, if needed. You can check in text files, HTML or other markup languages, nicely-formatted Word or RTF documents, or binaries such as JPGs or PDFs. (note displaying document differences doesn't work for binaries). Whatever anyone sends you, you download, or even scan or photograph, can be checked in to your document store. Other stuff you should "file" whether physically or electronically can include: ...

Documenting your test plans and the detailed procedures for each test is extremely important for making sure that your tests are effective, your results are reproducible, and you generate information that is as useful as possible in diagnosing any failures. Tests should be written in painfully clear language, even more so if they are going to get translated, or interpreted by non-native speakers of the language they're written in! Simplify, simplify, and avoid idioms, jokes, and flowery phrases. You can start a blog if you need room for your creative writing! If you do notice anyone has misunderstood the document, this is feedback you should use to make it even more straightforward. Be extremely clear about every detail; for example, if your device has a display and the test fixture has a display, be explicit about where the error message should appear. Use your judgment in deciding what should be in the test plan. It is definitely possible for test plans to get out of han...

Thanks to: Wikipedia.org user Acefitt The Boeing 737 MAX and its MCAS system (which tries to emulate the handling characteristics of earlier generations of the product) have been in the news a lot lately. Investigations are still ongoing, but the product is being blamed by many for two crashes which have resulted in the loss of almost 340 lives. That's a big quality problem. Clearly, the pressures on the Boeing are immense. The worldwide commercial aircraft market is over $191 billion , and dominated by cut-throat competition between Boeing and Airbus. The 737 has been in production since 1967, and if you are a pilot who was qualified to fly it in 1967, you still are today , despite vast numbers of changes to the aircraft. This is because Boeing has managed to keep the aircraft handling similar enough over time that the FAA (and other worldwide governmental authorities) has been willing to allow the "type certification" of pilots to be valid for all variants of t...

Documentation is the bread and butter of a QA engineer. We depend on it, and if it doesn't exist already, we may have to make it ourselves. Specifications Would you be surprised to hear that I've talked to more than one company who claimed they didn't have a specification for their product? Ridiculous, right? How did they know what to build? Of course they *did* have a specification, kind of, although they didn't realize it. It wasn't formalized, and this led to real problems. Companies have a variety of ways to write specifications, even if they aren't called that. They may have Marketing Requirements Documents, Product Requirements Documents, End User Safety documents, kickoff meetings, or other processes that are used to decide how the product needs to work. Often the MRD and/or PRD are handed to an engineering team, who then writes a "technical spec" based on their mechanical and electrical capabilities. Or maybe everyone just all sat a...

I hate to start the blog off on a downer note, but some important lessons here about personal integrity (and personal liability) in the field of QA. A QA engineer working for a supplier to SpaceX and other DoD suppliers was found to have forged signatures and stamps of inspectors of the parts his company (PMI Industries of Gates, NY) was selling to a number of rocket and missile manufacturers. In the field of QA, we're often subject to immense pressures from our management chain as well as other managers and directors in the company to "just sign off" on a product or test result. Once the CFO of a company I was working for cornere d me when nobody was around just to let me know that if we didn't ship the product soon, we wouldn't be able to make payroll! Talk about pressure! I did work harder to complete the factory test fixture I was building, but as the article indicates, if you go too far under pressure and falsify tests, you can open yourself up to ...

Welcome to the Startup Quality Blog; a blog about product quality for engineers, entrepreneurs, and managers! I've been involved in product quality since the early 2000s in the San Francisco bay area, working for a number of tech companies, large and small. My goal for the blog is to post about quality-related issues in the news, as well as deep-dives and interviews on quality-related topics. I'd especially like to feature posts on the tools and techniques in a QA engineer's metaphorical toolbox, and how they can benefit companies where quality responsibilities are currently shouldered by design engineers, managers, or users. I decided to call this the Startup Quality Blog because I feel like not only are these the companies that can often benefit from additional information and community around their product quality, but they are also the companies that can most easily make changes and take advantages of the best practices and institutional learning that a good underst...