Thursday, January 10, 2008

The Test Case in a Bug Report

Operating system and hardware The platform (operating system and hardware) indicates which system will reproduce the problem. If the behavior occurs on more than one platform, pick one to remove any ambiguity.
Source code for an executable test case The source code should be a stand-alone application that can compile without any external libraries. Again, shorter is better: Anything longer than 20 lines makes it more difficult for engineers to find the actual cause and increases the probability that the problem is actually in the bug submitter's code. Although extracting a small test case from a gigantic deployed application is a challenge, submit as short a test case as possible. This exercise may reveal that the problem is in one's own code, which means that a solution is immediately available.
Steps to reproduce Include the command-line invocation, even if it seems obvious.
Expected result The expected behavior tells us how the test case should behave if no bug were present. If the test case produces this expected result, we know that a fix works.
Actual result The actual behavior tells us the current results of the test case. If Sun engineering does not see this exact behavior, then the system configuration may be causing the problem.

No comments: