Testing versus Checking
I stumble upon this blog post Testing vs. Checking by Michael Bolton which I found very interesting and a fundamental for software testers. I tried to summarize it and place it in a table to see the distinction clearly.
|confirming existing beliefs||finding new information|
|a process of confirmation, verification, and validation||a process of exploration, discovery, investigation, and learning|
|focused on making sure that the program doesn’t fail||focused on “learning sufficiently everything that matters about how the program works and about how it might not work.” – James Bach|
|We are checking when:
* we’ve made a change to the code and we want to make sure that everything that worked before still works. When we have an assumption that’s important, we check to make sure the assumption holds.
* developers write and modify their code, creating automated routines that they run frequently to check to make sure that the code hasn’t broken
|We are testing when:
* we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn’t anticipated
* when we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before
|A check provides a binary result—true or false, yes or no||A test has an open-ended result|
|is all about asking and answering the question, “Does this assertion pass or fail?”||is about asking and answering the question “Is there a problem here?”|
|checking can prove the presence of bugs, but not their absence||the process of finding out whether our checks have been good enough|
|machine-decidable and are, in and of themselves, value-neutral||requires human, which makes it a sapient process|
|when done by a programmer, is mostly a quality assurance practice. The quality assurance angle: a programmer helps to assure the quality of his work by checking it.||is not quality assurance, but acts in service to it; we supply information to programmers and managers who have the authority to make decisions about the project.|
|A person who needs a clear, complete, up-to-date, unambiguous specification to proceed||Does not require the certainty of a perfect specification to make useful observations and inferences about the product|
|A person who needs a test script to proceed||His job is to discover information; often that information is in terms of inconsistencies between what people think and what’s true in reality|
|A person who does nothing but to compare a program against some reference||A tester’s task might be to reveal problems that occur when our excellent code calls buggy code in someone else’s library, for which we don’t have a specification|
I also liked how James Bach describes testing, “questioning a product in order to evaluate it.” Evaluation of a product doesn’t assure its quality, but it can inform decisions that will have an impact on quality.