The bug needs to be communicated and assigned to developers that can fix it. After the problem is resolved, fixes should be re-tested, and determinations made regarding requirements for regression testing to check that fixes didn't create problems elsewhere. If a problem-tracking system is in place, it should encapsulate these processes. A variety of commercial problem-tracking/management software tools are available (see the 'Tools' section for web resources with listings of such tools). The following are items to consider in the tracking process:
•Complete information such that developers can understand the bug, get an idea of it's severity, and reproduce it if necessary.
•Bug identifier (number, ID, etc.)
•Current bug status (e.g., 'Released for Retest', 'New', etc.)
•The application name or identifier and version
•The function, module, feature, object, screen, etc. where the bug occurred
•Environment specifics, system, platform, relevant hardware specifics
•Test case name/number/identifier
•One-line bug description
•Full bug description
•Description of steps needed to reproduce the bug if not covered by a test case or if the developer doesn't have easy access to the test case/test script/test tool
•Names and/or descriptions of file/data/messages/etc. used in test
•File excerpts/error messages/log file excerpts/screen shots/test tool logs that would be helpful in finding the cause of the problem
•Severity estimate (a 5-level range such as 1-5 or 'critical'-to-'low' is common)
•Was the bug reproducible?
•Tester name
•Test date
•Bug reporting date
•Name of developer/group/organization the problem is assigned to
•Description of problem cause
•Description of fix
•Code section/file/module/class/method that was fixed
•Date of fix
•Application version that contains the fix
•Tester responsible for retest
•Retest date
•Retest results
•Tester responsible for regression tests
•Regression testing results
Showing posts with label Software Testing. Show all posts
Showing posts with label Software Testing. Show all posts
Sunday, January 13, 2008
Saturday, January 12, 2008
What are the different types of Bugs we normally see in any of the Project?
1. What are the different types of Bugs we normally see in any of the Project? Include the severity as well.
The Life Cycle of a bug in general context is:
Bugs are usually logged by the development team (While Unit Testing) and also by testers (While sytem or other type of testing).
So let me explain in terms of a tester's perspective:
A tester finds a new defect/bug, so using a defect tracking tool logs it.
1. Its status is 'NEW' and assigns to the respective dev team (Team lead or Manager). 2. Th
e team lead assign's it to the team member, so the status is 'ASSIGNED TO'
3. The developer works on the bug fixes it and re-assings to the tester for testing. Now the status is 'RE-ASSIGNED'
4. The tester, check if the defect is fixed, if its fixed he changes the status to 'VERIFIED'
5. If the tester has the autority (depends on the company) he can after verifying change the status to 'FIXED'. If not the test lead can verify it and change the status to 'fixed'.
6. If the defect is not fixed he re-assign's the defect back to the dev team for re-fixing.
This is the life cycle of a bug.
1. User Interface Defects - Low
2. Boundary Related Defects - Medium
3. Error Handling Defects - Medium
4. Calculation Defects - High
5. Improper Service Levels (Control flow defects) - High
6. Interpreting Data Defects - High
7. Race Conditions (Compatibility and Intersystem defects)- High
8. Load Conditions (Memory Leakages under load) - High
9. Hardware Failures:- High
The Life Cycle of a bug in general context is:
Bugs are usually logged by the development team (While Unit Testing) and also by testers (While sytem or other type of testing).
So let me explain in terms of a tester's perspective:
A tester finds a new defect/bug, so using a defect tracking tool logs it.
1. Its status is 'NEW' and assigns to the respective dev team (Team lead or Manager). 2. Th
e team lead assign's it to the team member, so the status is 'ASSIGNED TO'
3. The developer works on the bug fixes it and re-assings to the tester for testing. Now the status is 'RE-ASSIGNED'
4. The tester, check if the defect is fixed, if its fixed he changes the status to 'VERIFIED'
5. If the tester has the autority (depends on the company) he can after verifying change the status to 'FIXED'. If not the test lead can verify it and change the status to 'fixed'.
6. If the defect is not fixed he re-assign's the defect back to the dev team for re-fixing.
This is the life cycle of a bug.
1. User Interface Defects - Low
2. Boundary Related Defects - Medium
3. Error Handling Defects - Medium
4. Calculation Defects - High
5. Improper Service Levels (Control flow defects) - High
6. Interpreting Data Defects - High
7. Race Conditions (Compatibility and Intersystem defects)- High
8. Load Conditions (Memory Leakages under load) - High
9. Hardware Failures:- High
Friday, January 11, 2008
Top Ten Tips for Bug Tracking !!!!
1. A good tester will always try to reduce the repro steps to the minimal steps to reproduce; this is extremely helpful for the programmer who has to find the bug.
2. Remember that the only person who can close a bug is the person who opened it in the first place. Anyone can resolve it, but only the person who saw the bug can really be sure that what they saw is fixed.
3. There are many ways to resolve a bug. FogBUGZ allows you to resolve a bug as fixed, won't fix, postponed, not repro, duplicate, or by design.
4. Not Repro means that nobody could ever reproduce the bug. Programmers often use this when the bug report is missing the repro steps.
5. You'll want to keep careful track of versions. Every build of the software that you give to testers should have a build ID number so that the poor tester doesn't have to retest the bug on a version of the software where it wasn't even supposed to be fixed.
6. If you're a programmer, and you're having trouble getting testers to use the bug database, just don't accept bug reports by any other method. If your testers are used to sending you email with bug reports, just bounce the emails back to them with a brief message: "please put this in the bug database. I can't keep track of emails."
7. If you're a tester, and you're having trouble getting programmers to use the bug database, just don't tell them about bugs - put them in the database and let the database email them.
8. If you're a programmer, and only some of your colleagues use the bug database, just start assigning them bugs in the database. Eventually they'll get the hint.
9. If you're a manager, and nobody seems to be using the bug database that you installed at great expense, start assigning new features to people using bugs. A bug database is also a great "unimplemented feature" database, too.
10. Avoid the temptation to add new fields to the bug database. Every month or so, somebody will come up with a great idea for a new field to put in the database. You get all kinds of clever ideas, for example, keeping track of the file where the bug was found; keeping track of what % of the time the bug is reproducible; keeping track of how many times the bug occurred; keeping track of which exact versions of which DLLs were installed on the machine where the bug happened. It's very important not to give in to these ideas. If you do, your new bug entry screen will end up with a thousand fields that you need to supply, and nobody will want to input bug reports any more. For the bug database to work, everybody needs to use it, and if entering bugs "formally" is too much work, people will go around the bug database.
2. Remember that the only person who can close a bug is the person who opened it in the first place. Anyone can resolve it, but only the person who saw the bug can really be sure that what they saw is fixed.
3. There are many ways to resolve a bug. FogBUGZ allows you to resolve a bug as fixed, won't fix, postponed, not repro, duplicate, or by design.
4. Not Repro means that nobody could ever reproduce the bug. Programmers often use this when the bug report is missing the repro steps.
5. You'll want to keep careful track of versions. Every build of the software that you give to testers should have a build ID number so that the poor tester doesn't have to retest the bug on a version of the software where it wasn't even supposed to be fixed.
6. If you're a programmer, and you're having trouble getting testers to use the bug database, just don't accept bug reports by any other method. If your testers are used to sending you email with bug reports, just bounce the emails back to them with a brief message: "please put this in the bug database. I can't keep track of emails."
7. If you're a tester, and you're having trouble getting programmers to use the bug database, just don't tell them about bugs - put them in the database and let the database email them.
8. If you're a programmer, and only some of your colleagues use the bug database, just start assigning them bugs in the database. Eventually they'll get the hint.
9. If you're a manager, and nobody seems to be using the bug database that you installed at great expense, start assigning new features to people using bugs. A bug database is also a great "unimplemented feature" database, too.
10. Avoid the temptation to add new fields to the bug database. Every month or so, somebody will come up with a great idea for a new field to put in the database. You get all kinds of clever ideas, for example, keeping track of the file where the bug was found; keeping track of what % of the time the bug is reproducible; keeping track of how many times the bug occurred; keeping track of which exact versions of which DLLs were installed on the machine where the bug happened. It's very important not to give in to these ideas. If you do, your new bug entry screen will end up with a thousand fields that you need to supply, and nobody will want to input bug reports any more. For the bug database to work, everybody needs to use it, and if entering bugs "formally" is too much work, people will go around the bug database.
Thursday, January 10, 2008
How to Write a Helpful Bug Report
Imagine a customer telling you that your product is not working. You would probably have many questions:
How serious is the situation?
Is there a defect in your product, or is the defect in another part of the customer system?
Is the issue already known?
Is a solution already available?
Now imagine thousands of such incidents from around the world. How would you analyze these incidents as quickly, efficiently, and accurately as possible? After all, each report is a chance to make the product better for everyone.
This scenario is the challenge that the WebBugs team faces every day. Ideally, the team would be able to investigate every bug report in person. But this approach is not realistic, so we have designed a bug-reporting site to help us gather the information we need.
Our goal is to assist developers in creating a report that is complete and compact. A complete report allows our team's developers to reproduce the problem locally. A compact report helps us focus the investigation so that we can quickly determine the cause of the behavior.
Let's tour the bug-reporting site to see how we pursue this goal. And note that this document applies only to bugs. It does not discuss the trail for reporting documentation issues and requests for enhancements, which are other options on the bug-reporting site.
How serious is the situation?
Is there a defect in your product, or is the defect in another part of the customer system?
Is the issue already known?
Is a solution already available?
Now imagine thousands of such incidents from around the world. How would you analyze these incidents as quickly, efficiently, and accurately as possible? After all, each report is a chance to make the product better for everyone.
This scenario is the challenge that the WebBugs team faces every day. Ideally, the team would be able to investigate every bug report in person. But this approach is not realistic, so we have designed a bug-reporting site to help us gather the information we need.
Our goal is to assist developers in creating a report that is complete and compact. A complete report allows our team's developers to reproduce the problem locally. A compact report helps us focus the investigation so that we can quickly determine the cause of the behavior.
Let's tour the bug-reporting site to see how we pursue this goal. And note that this document applies only to bugs. It does not discuss the trail for reporting documentation issues and requests for enhancements, which are other options on the bug-reporting site.
The Test Case in a Bug Report
Operating system and hardware The platform (operating system and hardware) indicates which system will reproduce the problem. If the behavior occurs on more than one platform, pick one to remove any ambiguity.
Source code for an executable test case The source code should be a stand-alone application that can compile without any external libraries. Again, shorter is better: Anything longer than 20 lines makes it more difficult for engineers to find the actual cause and increases the probability that the problem is actually in the bug submitter's code. Although extracting a small test case from a gigantic deployed application is a challenge, submit as short a test case as possible. This exercise may reveal that the problem is in one's own code, which means that a solution is immediately available.
Steps to reproduce Include the command-line invocation, even if it seems obvious.
Expected result The expected behavior tells us how the test case should behave if no bug were present. If the test case produces this expected result, we know that a fix works.
Actual result The actual behavior tells us the current results of the test case. If Sun engineering does not see this exact behavior, then the system configuration may be causing the problem.
Source code for an executable test case The source code should be a stand-alone application that can compile without any external libraries. Again, shorter is better: Anything longer than 20 lines makes it more difficult for engineers to find the actual cause and increases the probability that the problem is actually in the bug submitter's code. Although extracting a small test case from a gigantic deployed application is a challenge, submit as short a test case as possible. This exercise may reveal that the problem is in one's own code, which means that a solution is immediately available.
Steps to reproduce Include the command-line invocation, even if it seems obvious.
Expected result The expected behavior tells us how the test case should behave if no bug were present. If the test case produces this expected result, we know that a fix works.
Actual result The actual behavior tells us the current results of the test case. If Sun engineering does not see this exact behavior, then the system configuration may be causing the problem.
Bug Life Cycle
In software development process, the bug has a life cycle. The bug should go through the life cycle to be closed. A specific life cycle ensures that the process is standardized. The bug attains different states in the life cycle. The life cycle of the bug can be shown diagrammatically as follows:
The different states of a bug can be summarized as follows:
1. New
2. Open
3. Assign
4. Test
5. Verified
6. Deferred
7. Reopened
8. Duplicate
9. Rejected and
10. Closed
Description of Various Stages:
1. New: When the bug is posted for the first time, its state will be “NEW”. This means that the bug is not yet approved.
2. Open: After a tester has posted a bug, the lead of the tester approves that the bug is genuine and he changes the state as “OPEN”.
3. Assign: Once the lead changes the state as “OPEN”, he assigns the bug to corresponding developer or developer team. The state of the bug now is changed to “ASSIGN”.
4. Test: Once the developer fixes the bug, he has to assign the bug to the testing team for next round of testing. Before he releases the software with bug fixed, he changes the state of bug to “TEST”. It specifies that the bug has been fixed and is released to testing team.
5. Deferred: The bug, changed to deferred state means the bug is expected to be fixed in next releases. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software.
6. Rejected: If the developer feels that the bug is not genuine, he rejects the bug. Then the state of the bug is changed to “REJECTED”.
7. Duplicate: If the bug is repeated twice or the two bugs mention the same concept of the bug, then one bug status is changed to “DUPLICATE”.
8. Verified: Once the bug is fixed and the status is changed to “TEST”, the tester tests the bug. If the bug is not present in the software, he approves that the bug is fixed and changes the status to “VERIFIED”.
9. Reopened: If the bug still exists even after the bug is fixed by the developer, the tester changes the status to “REOPENED”. The bug traverses the life cycle once again.
10. Closed: Once the bug is fixed, it is tested by the tester. If the tester feels that the bug no longer exists in the software, he changes the status of the bug to “CLOSED”. This state means that the bug is fixed, tested and approved.
While defect prevention is much more effective and efficient in reducing the number of defects, most organization conducts defect discovery and removal. Discovering and removing defects is an expensive and inefficient process. It is much more efficient for an organization to conduct activities that prevent defects.
The different states of a bug can be summarized as follows:
1. New
2. Open
3. Assign
4. Test
5. Verified
6. Deferred
7. Reopened
8. Duplicate
9. Rejected and
10. Closed
Description of Various Stages:
1. New: When the bug is posted for the first time, its state will be “NEW”. This means that the bug is not yet approved.
2. Open: After a tester has posted a bug, the lead of the tester approves that the bug is genuine and he changes the state as “OPEN”.
3. Assign: Once the lead changes the state as “OPEN”, he assigns the bug to corresponding developer or developer team. The state of the bug now is changed to “ASSIGN”.
4. Test: Once the developer fixes the bug, he has to assign the bug to the testing team for next round of testing. Before he releases the software with bug fixed, he changes the state of bug to “TEST”. It specifies that the bug has been fixed and is released to testing team.
5. Deferred: The bug, changed to deferred state means the bug is expected to be fixed in next releases. The reasons for changing the bug to this state have many factors. Some of them are priority of the bug may be low, lack of time for the release or the bug may not have major effect on the software.
6. Rejected: If the developer feels that the bug is not genuine, he rejects the bug. Then the state of the bug is changed to “REJECTED”.
7. Duplicate: If the bug is repeated twice or the two bugs mention the same concept of the bug, then one bug status is changed to “DUPLICATE”.
8. Verified: Once the bug is fixed and the status is changed to “TEST”, the tester tests the bug. If the bug is not present in the software, he approves that the bug is fixed and changes the status to “VERIFIED”.
9. Reopened: If the bug still exists even after the bug is fixed by the developer, the tester changes the status to “REOPENED”. The bug traverses the life cycle once again.
10. Closed: Once the bug is fixed, it is tested by the tester. If the tester feels that the bug no longer exists in the software, he changes the status of the bug to “CLOSED”. This state means that the bug is fixed, tested and approved.
While defect prevention is much more effective and efficient in reducing the number of defects, most organization conducts defect discovery and removal. Discovering and removing defects is an expensive and inefficient process. It is much more efficient for an organization to conduct activities that prevent defects.
Bug Tracking Software Features
Do you know which bug tracking software is best for your development environment? Besides the obvious issues of compatibility, do you know what other features are important? Don’t let the sales pitches confuse. Before you purchase bug tracking software, know what your software developers and software testers want and need. After all, they’re the ones who will be using this tool most.
Reliable bug tracking software is itself bug-free and stable. Select a product that has a proven track record of success. If possible, talk with actual users to see whether or not the software is meeting their needs.
You want your bug tracking software to be customizable so that it works the way your development team works. But you don’t want this task to be so difficult that you have to first take a course to learn how to accomplish this. A good interface is simple and straightforward, and at the same time, intuitive.
Select bug tracking software that fits your budget and works on the hardware configuration you already have in place. If it requires additional, costly hardware to operate, keep looking.
As for features, including the ability to prioritize a reported bug helps managers allocate resources more efficiently. It’s important to know additional information about each reported bug, and effective bug tracking software facilitates adding notes. For example, it’s important to reproduce the steps that led to the discovery of the bug, the actions that should have happened when the bug was discovered, the actions that actually did happen, and the version that was used at the time of discrepancy. As the bug works its way from team member to team member, the good software tracks that history and allows additional notes to be appended at any time.
Reliable bug tracking software doubles as a management tool. It produces useful reports including work that has already been assigned to each team member. Such a report helps management reprioritize and reassign work when necessary. Also important is a built-in method for notifying specific team members that new work has been assigned to them. As each team member completes work on the bug, they need the ability to update the status of the bug. The bug itself needs to be reassigned to another individual or marked as closed.
Be sure that the bug tracking software you select allows customization of the steps necessary to close an open bug report. Also make sure the software allows you to designate the person(s) who has the authority to close a bug report.
Reliable bug tracking software is itself bug-free and stable. Select a product that has a proven track record of success. If possible, talk with actual users to see whether or not the software is meeting their needs.
You want your bug tracking software to be customizable so that it works the way your development team works. But you don’t want this task to be so difficult that you have to first take a course to learn how to accomplish this. A good interface is simple and straightforward, and at the same time, intuitive.
Select bug tracking software that fits your budget and works on the hardware configuration you already have in place. If it requires additional, costly hardware to operate, keep looking.
As for features, including the ability to prioritize a reported bug helps managers allocate resources more efficiently. It’s important to know additional information about each reported bug, and effective bug tracking software facilitates adding notes. For example, it’s important to reproduce the steps that led to the discovery of the bug, the actions that should have happened when the bug was discovered, the actions that actually did happen, and the version that was used at the time of discrepancy. As the bug works its way from team member to team member, the good software tracks that history and allows additional notes to be appended at any time.
Reliable bug tracking software doubles as a management tool. It produces useful reports including work that has already been assigned to each team member. Such a report helps management reprioritize and reassign work when necessary. Also important is a built-in method for notifying specific team members that new work has been assigned to them. As each team member completes work on the bug, they need the ability to update the status of the bug. The bug itself needs to be reassigned to another individual or marked as closed.
Be sure that the bug tracking software you select allows customization of the steps necessary to close an open bug report. Also make sure the software allows you to designate the person(s) who has the authority to close a bug report.
Why does software have bugs?
Miscommunication or no communication - as to specifics of what an application should or shouldn't do (the application's requirements).
* Software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windows-type interfaces, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. And the use of object-oriented techniques can complicate instead of simplify a project unless it is well engineered.
* Programming errors - programmers, like anyone else, can make mistakes.
* Changing requirements - the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of keeping track of changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control.
* time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
* egos - people prefer to say things like:
* 'no problem'
* 'piece of cake'
* 'I can whip that out in a few hours'
* 'it should be easy to update that old code'
* instead of:
* 'that adds a lot of complexity and we could end up
* making a lot of mistakes'
* 'we have no idea if we can do that; we'll wing it'
* 'I can't estimate how long it will take, until I
* take a close look at it'
* 'we can't figure out what that old spaghetti code
* did in the first place'
* If there are too many unrealistic 'no problem's', the result is bugs.
* poorly documented code - it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read').
* software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.
Post Resume: Click here to Upload your Resume & Apply for Jobs
* Software complexity - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windows-type interfaces, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. And the use of object-oriented techniques can complicate instead of simplify a project unless it is well engineered.
* Programming errors - programmers, like anyone else, can make mistakes.
* Changing requirements - the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of keeping track of changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control.
* time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
* egos - people prefer to say things like:
* 'no problem'
* 'piece of cake'
* 'I can whip that out in a few hours'
* 'it should be easy to update that old code'
* instead of:
* 'that adds a lot of complexity and we could end up
* making a lot of mistakes'
* 'we have no idea if we can do that; we'll wing it'
* 'I can't estimate how long it will take, until I
* take a close look at it'
* 'we can't figure out what that old spaghetti code
* did in the first place'
* If there are too many unrealistic 'no problem's', the result is bugs.
* poorly documented code - it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read').
* software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.
Post Resume: Click here to Upload your Resume & Apply for Jobs
Debugging approach !!!!
In general, three categories for debugging approaches may be proposed.
• Brute force
• Back tracking
• Cause elimination
The brute force category of debugging is probably the most common and efficient method for isolating the cause of a software error. Brute force debugging methods are applied when all methods of debugging fail. Using a philosophy, memory dumps are taken, run time traces are invoked and the program is loaded with WRITE statement. When this is done, one finds a clue by the information produced which leads to cause of an error.
Backtracking is a common debugging approach that can be used successfully in small programs. Beginning at the site where a symptom has been uncovered, the source code is traced backward (manually) until the site of the cause is found. This process has a limitation when the source lines are more.
Cause Elimination is manifested by induction or deduction and introduces the concept of binary partitioning. Data related to the error occurrence are organized to isolate potential causes.
Alternatively, a list of all possible causes is developed and tests are conducted to eliminate each.
If initial tests indicate that a particular cause hypothesis shows promise the data are refined in an attempt to isolate the bug.
• Brute force
• Back tracking
• Cause elimination
The brute force category of debugging is probably the most common and efficient method for isolating the cause of a software error. Brute force debugging methods are applied when all methods of debugging fail. Using a philosophy, memory dumps are taken, run time traces are invoked and the program is loaded with WRITE statement. When this is done, one finds a clue by the information produced which leads to cause of an error.
Backtracking is a common debugging approach that can be used successfully in small programs. Beginning at the site where a symptom has been uncovered, the source code is traced backward (manually) until the site of the cause is found. This process has a limitation when the source lines are more.
Cause Elimination is manifested by induction or deduction and introduces the concept of binary partitioning. Data related to the error occurrence are organized to isolate potential causes.
Alternatively, a list of all possible causes is developed and tests are conducted to eliminate each.
If initial tests indicate that a particular cause hypothesis shows promise the data are refined in an attempt to isolate the bug.
Defect Tracking by Test Lead !!!
The test lead, categorizes the defects after meetings with the clients as,
Modify Cases: Test cases to be modified. This may arise when the testers understanding may be incorrect.
Discussion Items: Arises when there is a difference of opinion between the test and the development team. This is marked to the Domain consultant for final verdict.
Change Technology: Arises when the development team has to fix the bug.
Data Related: Arises when the defect is due to data and not coding.
User Training: Arises when the defect is not severe or technically not feasible to fix, it is decided to train the user on the defect. This should ideally not be critical.
New Requirement: Inclusion of functionality after discussion
User Maintenance: Masters and Parameter maintained by the user causing the defect.
Observation: Any other observation, which is not classified in the above categories like a user perspective GUI defect.
Reporting is done for defect evaluation and also to ensure that the development team is aware of the defects found and is in the process of resolving the defects. A detailed report of the defects is generated everyday and given to the development team for their feedback on defect resolution. A summary report is generated for every report to evaluate the rate at which new defects are found and the rate at which the defects are tracked to closure.
Defect counts are reported as a function of time, creating a Defect Trend diagram or report, and as a function of one or more defect parameters like category or status, creating a Defect Density report. These types of analysis provide a perspective on the trends or distribution of defects that reveal the system’s reliability, respectively.
It is expected that defect discovery rates will eventually diminish as the testing and fixing progresses. A threshold can be established below which the system can be deployed. Defect counts can also be reported based on the origin in the implementation model, allowing detection of “weak modules”, “hot spots”, parts of the system that are fixed again and again, indicating some fundamental design flaw.
Defects included in an analysis of this kind are confirmed defects. Not all reported defects report an actual flaw, as some may be enhancement requests, out of the scope of the system, or describe an already reported defect. However, there is a value to looking at and analysing why there are many defects being reported that are either duplicates or not confirmed defects.
Modify Cases: Test cases to be modified. This may arise when the testers understanding may be incorrect.
Discussion Items: Arises when there is a difference of opinion between the test and the development team. This is marked to the Domain consultant for final verdict.
Change Technology: Arises when the development team has to fix the bug.
Data Related: Arises when the defect is due to data and not coding.
User Training: Arises when the defect is not severe or technically not feasible to fix, it is decided to train the user on the defect. This should ideally not be critical.
New Requirement: Inclusion of functionality after discussion
User Maintenance: Masters and Parameter maintained by the user causing the defect.
Observation: Any other observation, which is not classified in the above categories like a user perspective GUI defect.
Reporting is done for defect evaluation and also to ensure that the development team is aware of the defects found and is in the process of resolving the defects. A detailed report of the defects is generated everyday and given to the development team for their feedback on defect resolution. A summary report is generated for every report to evaluate the rate at which new defects are found and the rate at which the defects are tracked to closure.
Defect counts are reported as a function of time, creating a Defect Trend diagram or report, and as a function of one or more defect parameters like category or status, creating a Defect Density report. These types of analysis provide a perspective on the trends or distribution of defects that reveal the system’s reliability, respectively.
It is expected that defect discovery rates will eventually diminish as the testing and fixing progresses. A threshold can be established below which the system can be deployed. Defect counts can also be reported based on the origin in the implementation model, allowing detection of “weak modules”, “hot spots”, parts of the system that are fixed again and again, indicating some fundamental design flaw.
Defects included in an analysis of this kind are confirmed defects. Not all reported defects report an actual flaw, as some may be enhancement requests, out of the scope of the system, or describe an already reported defect. However, there is a value to looking at and analysing why there are many defects being reported that are either duplicates or not confirmed defects.
Types of Defects
Defects that are detected by the tester are classified into categories by the nature of the defect. The following are the classification
Showstopper (X): The impact of the defect is severe and the system cannot go into the production environment without resolving the defect since an interim solution may not be available.
Critical (C): The impact of the defect is severe, however an interim solution is available. The defect should not hinder the test process in any way.
Non critical (N): All defects that are not in the X or C category are deemed to be in the N category. These are also the defects that could potentially be resolved via documentation and user training. These can be Graphic User Interface (GUI) defects are some minor field level observations.
Showstopper (X): The impact of the defect is severe and the system cannot go into the production environment without resolving the defect since an interim solution may not be available.
Critical (C): The impact of the defect is severe, however an interim solution is available. The defect should not hinder the test process in any way.
Non critical (N): All defects that are not in the X or C category are deemed to be in the N category. These are also the defects that could potentially be resolved via documentation and user training. These can be Graphic User Interface (GUI) defects are some minor field level observations.
What is the difference between exception and validation testing?
Validation testing aims to demonstrate that the software functions in a manner that can be reasonably expected by the customer. Testing the software in conformance to the Software Requirements Specifications.
Exception testing deals with handling the exceptions (unexpected events) while the AUT is run. Basically this testing involves how to change the control flow of the AUT when an exception arises
Exception testing deals with handling the exceptions (unexpected events) while the AUT is run. Basically this testing involves how to change the control flow of the AUT when an exception arises
How can it be known when to stop Testing?
This can be difficult to determine. Many modern software applications are so complex, and run in such an interdependent environment, that complete testing can never be done. Common factors in deciding when to stop are:
• Deadlines (release deadlines, testing deadlines, etc.)
• Test cases completed with certain percentage passed
• Test budget depleted
• Coverage of code/functionality/requirements reaches a specified point
• Bug rate falls below a certain level
• Beta or alpha testing period ends
• Deadlines (release deadlines, testing deadlines, etc.)
• Test cases completed with certain percentage passed
• Test budget depleted
• Coverage of code/functionality/requirements reaches a specified point
• Bug rate falls below a certain level
• Beta or alpha testing period ends
Bug Reports - Other Considerations
* If your bug is randomly reproducible, just mention it in your bug report. But don.t forget to file it. You can always add the exact steps to reproduce anytime later you (or anyone else) discover them. This will also come to your rescue when someone else reports this issue, especially if it.s a serious one.
* Mention the error messages in the bug report, especially if they are numbered. For example, error messages from the database.
* Mention the version numbers and build numbers in the bug reports.
* Mention the platforms on which the issue is reproducible. Precisely mention the platforms on which the issue is not reproducible. Also understand that there is difference between the issue being not reproducible on a particular platform and it not being tested on that platform. This might lead to confusion.
* If you come across several problems having the same cause, write a single bug report. The fix of the problem will be only one. Similarly, if you come across similar problems at different locations requiring the same kind of fix but at different places, write separate bug reports for each of the problems. One bug report for only one fix.
* If the test environment on which the bug is reproducible is accessible to the developers, mention the details of accessing this setup. This will help them save time to setting up the environment to reproduce your bug.
* Under no circumstances should you hold on to any information regarding the bug. Unnecessary iterations of the bug report between the developer and the tester before being fixed is just waste of time due to ineffective bug reporting.
* Mention the error messages in the bug report, especially if they are numbered. For example, error messages from the database.
* Mention the version numbers and build numbers in the bug reports.
* Mention the platforms on which the issue is reproducible. Precisely mention the platforms on which the issue is not reproducible. Also understand that there is difference between the issue being not reproducible on a particular platform and it not being tested on that platform. This might lead to confusion.
* If you come across several problems having the same cause, write a single bug report. The fix of the problem will be only one. Similarly, if you come across similar problems at different locations requiring the same kind of fix but at different places, write separate bug reports for each of the problems. One bug report for only one fix.
* If the test environment on which the bug is reproducible is accessible to the developers, mention the details of accessing this setup. This will help them save time to setting up the environment to reproduce your bug.
* Under no circumstances should you hold on to any information regarding the bug. Unnecessary iterations of the bug report between the developer and the tester before being fixed is just waste of time due to ineffective bug reporting.
Debugger Commands
The debugger recognizes the following commands. Most commands can be abbreviated to one or two letters; e.g. "h(elp)" means that either "h" or "help" can be used to enter the help command (but not "he" or "hel", nor "H" or "Help" or "HELP"). Arguments to commands must be separated by whitespace (spaces or tabs). Optional arguments are enclosed in square brackets ("[]") in the command syntax; the square brackets must not be typed. Alternatives in the command syntax are separated by a vertical bar ("|").
Entering a blank line repeats the last command entered. Exception: if the last command was a "list" command, the next 11 lines are listed.
Commands that the debugger doesn't recognize are assumed to be Python statements and are executed in the context of the program being debugged. Python statements can also be prefixed with an exclamation point ("!"). This is a powerful way to inspect the program being debugged; it is even possible to change a variable or call a function. When an exception occurs in such a statement, the exception name is printed but the debugger's state is not changed.
Multiple commands may be entered on a single line, separated by ";;". (A single ";" is not used as it is the separator for multiple commands in a line that is passed to the Python parser.) No intelligence is applied to separating the commands; the input is split at the first ";;" pair, even if it is in the middle of a quoted string.
The debugger supports aliases. Aliases can have parameters which allows one a certain level of adaptability to the context under examination.
If a file .pdbrc exists in the user's home directory or in the current directory, it is read in and executed as if it had been typed at the debugger prompt. This is particularly useful for aliases. If both files exist, the one in the home directory is read first and aliases defined there can be overridden by the local file.
h(elp) [command]
Without argument, print the list of available commands. With a command as argument, print help about that command. "help pdb" displays the full documentation file; if the environment variable PAGER is defined, the file is piped through that command instead. Since the command argument must be an identifier, "help exec" must be entered to get help on the "!" command.
w(here)
Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands.
d(own)
Move the current frame one level down in the stack trace (to a newer frame).
u(p)
Move the current frame one level up in the stack trace (to an older frame).
b(reak) [[filename:]lineno|function[, condition]]
With a lineno argument, set a break there in the current file. With a function argument, set a break at the first executable statement within that function. The line number may be prefixed with a filename and a colon, to specify a breakpoint in another file (probably one that hasn't been loaded yet). The file is searched on sys.path. Note that each breakpoint is assigned a number to which all the other breakpoint commands refer.
If a second argument is present, it is an expression which must evaluate to true before the breakpoint is honored.
Without argument, list all breaks, including for each breakpoint, the number of times that breakpoint has been hit, the current ignore count, and the associated condition if any.
tbreak [[filename:]lineno|function[, condition]]
Temporary breakpoint, which is removed automatically when it is first hit. The arguments are the same as break.
cl(ear) [bpnumber [bpnumber ...]]
With a space separated list of breakpoint numbers, clear those breakpoints. Without argument, clear all breaks (but first ask confirmation).
disable [bpnumber [bpnumber ...]]
Disables the breakpoints given as a space separated list of breakpoint numbers. Disabling a breakpoint means it cannot cause the program to stop execution, but unlike clearing a breakpoint, it remains in the list of breakpoints and can be (re-)enabled.
enable [bpnumber [bpnumber ...]]
Enables the breakpoints specified.
ignore bpnumber [count]
Sets the ignore count for the given breakpoint number. If count is omitted, the ignore count is set to 0. A breakpoint becomes active when the ignore count is zero. When non-zero, the count is decremented each time the breakpoint is reached and the breakpoint is not disabled and any associated condition evaluates to true.
condition bpnumber [condition]
Condition is an expression which must evaluate to true before the breakpoint is honored. If condition is absent, any existing condition is removed; i.e., the breakpoint is made unconditional.
commands [bpnumber]
Specify a list of commands for breakpoint number bpnumber. The commands themselves appear on the following lines. Type a line containing just 'end' to terminate the commands. An example:
(Pdb) commands 1(com) print some_variable(com) end(Pdb)To remove all commands from a breakpoint, type commands and follow it immediately with end; that is, give no commands.
With no bpnumber argument, commands refers to the last breakpoint set.
You can use breakpoint commands to start your program up again. Simply use the continue command, or step, or any other command that resumes execution.
Specifying any command resuming execution (currently continue, step, next, return, jump, quit and their abbreviations) terminates the command list (as if that command was immediately followed by end). This is because any time you resume execution (even with a simple next or step), you may encounter· another breakpoint-which could have its own command list, leading to ambiguities about which list to execute.
If you use the 'silent' command in the command list, the usual message about stopping at a breakpoint is not printed. This may be desirable for breakpoints that are to print a specific message and then continue. If none of the other commands print anything, you see no sign that the breakpoint was reached.
New in version 2.5.
s(tep)
Execute the current line, stop at the first possible occasion (either in a function that is called or on the next line in the current function).
n(ext)
Continue execution until the next line in the current function is reached or it returns. (The difference between "next" and "step" is that "step" stops inside a called function, while "next" executes called functions at (nearly) full speed, only stopping at the next line in the current function.)
r(eturn)
Continue execution until the current function returns.
c(ont(inue))
Continue execution, only stop when a breakpoint is encountered.
j(ump) lineno
Set the next line that will be executed. Only available in the bottom-most frame. This lets you jump back and execute code again, or jump forward to skip code that you don't want to run.
It should be noted that not all jumps are allowed -- for instance it is not possible to jump into the middle of a for loop or out of a finally clause.
l(ist) [first[, last]]
List source code for the current file. Without arguments, list 11 lines around the current line or continue the previous listing. With one argument, list 11 lines around at that line. With two arguments, list the given range; if the second argument is less than the first, it is interpreted as a count.
a(rgs)
Print the argument list of the current function.
p expression
Evaluate the expression in the current context and print its value. Note: "print" can also be used, but is not a debugger command -- this executes the Python print statement.
pp expression
Like the "p" command, except the value of the expression is pretty-printed using the pprint module.
alias [name [command]]
Creates an alias called name that executes command. The command must not be enclosed in quotes. Replaceable parameters can be indicated by "%1", "%2", and so on, while "%*" is replaced by all the parameters. If no command is given, the current alias for name is shown. If no arguments are given, all aliases are listed.
Aliases may be nested and can contain anything that can be legally typed at the pdb prompt. Note that internal pdb commands can be overridden by aliases. Such a command is then hidden until the alias is removed. Aliasing is recursively applied to the first word of the command line; all other words in the line are left alone.
As an example, here are two useful aliases (especially when placed in the .pdbrc file):
#Print instance variables (usage "pi classInst")alias pi for k in %1.__dict__.keys(): print "%1.",k,"=",%1.__dict__[k]#Print instance variables in selfalias ps pi self
unalias name
Deletes the specified alias.
statement
Execute the (one-line) statement in the context of the current stack frame. The exclamation point can be omitted unless the first word of the statement resembles a debugger command. To set a global variable, you can prefix the assignment command with a "global" command on the same line, e.g.:
(Pdb) global list_options; list_options = ['-l'](Pdb)
q(uit)
Quit from the debugger. The program being executed is aborted.
Entering a blank line repeats the last command entered. Exception: if the last command was a "list" command, the next 11 lines are listed.
Commands that the debugger doesn't recognize are assumed to be Python statements and are executed in the context of the program being debugged. Python statements can also be prefixed with an exclamation point ("!"). This is a powerful way to inspect the program being debugged; it is even possible to change a variable or call a function. When an exception occurs in such a statement, the exception name is printed but the debugger's state is not changed.
Multiple commands may be entered on a single line, separated by ";;". (A single ";" is not used as it is the separator for multiple commands in a line that is passed to the Python parser.) No intelligence is applied to separating the commands; the input is split at the first ";;" pair, even if it is in the middle of a quoted string.
The debugger supports aliases. Aliases can have parameters which allows one a certain level of adaptability to the context under examination.
If a file .pdbrc exists in the user's home directory or in the current directory, it is read in and executed as if it had been typed at the debugger prompt. This is particularly useful for aliases. If both files exist, the one in the home directory is read first and aliases defined there can be overridden by the local file.
h(elp) [command]
Without argument, print the list of available commands. With a command as argument, print help about that command. "help pdb" displays the full documentation file; if the environment variable PAGER is defined, the file is piped through that command instead. Since the command argument must be an identifier, "help exec" must be entered to get help on the "!" command.
w(here)
Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands.
d(own)
Move the current frame one level down in the stack trace (to a newer frame).
u(p)
Move the current frame one level up in the stack trace (to an older frame).
b(reak) [[filename:]lineno|function[, condition]]
With a lineno argument, set a break there in the current file. With a function argument, set a break at the first executable statement within that function. The line number may be prefixed with a filename and a colon, to specify a breakpoint in another file (probably one that hasn't been loaded yet). The file is searched on sys.path. Note that each breakpoint is assigned a number to which all the other breakpoint commands refer.
If a second argument is present, it is an expression which must evaluate to true before the breakpoint is honored.
Without argument, list all breaks, including for each breakpoint, the number of times that breakpoint has been hit, the current ignore count, and the associated condition if any.
tbreak [[filename:]lineno|function[, condition]]
Temporary breakpoint, which is removed automatically when it is first hit. The arguments are the same as break.
cl(ear) [bpnumber [bpnumber ...]]
With a space separated list of breakpoint numbers, clear those breakpoints. Without argument, clear all breaks (but first ask confirmation).
disable [bpnumber [bpnumber ...]]
Disables the breakpoints given as a space separated list of breakpoint numbers. Disabling a breakpoint means it cannot cause the program to stop execution, but unlike clearing a breakpoint, it remains in the list of breakpoints and can be (re-)enabled.
enable [bpnumber [bpnumber ...]]
Enables the breakpoints specified.
ignore bpnumber [count]
Sets the ignore count for the given breakpoint number. If count is omitted, the ignore count is set to 0. A breakpoint becomes active when the ignore count is zero. When non-zero, the count is decremented each time the breakpoint is reached and the breakpoint is not disabled and any associated condition evaluates to true.
condition bpnumber [condition]
Condition is an expression which must evaluate to true before the breakpoint is honored. If condition is absent, any existing condition is removed; i.e., the breakpoint is made unconditional.
commands [bpnumber]
Specify a list of commands for breakpoint number bpnumber. The commands themselves appear on the following lines. Type a line containing just 'end' to terminate the commands. An example:
(Pdb) commands 1(com) print some_variable(com) end(Pdb)To remove all commands from a breakpoint, type commands and follow it immediately with end; that is, give no commands.
With no bpnumber argument, commands refers to the last breakpoint set.
You can use breakpoint commands to start your program up again. Simply use the continue command, or step, or any other command that resumes execution.
Specifying any command resuming execution (currently continue, step, next, return, jump, quit and their abbreviations) terminates the command list (as if that command was immediately followed by end). This is because any time you resume execution (even with a simple next or step), you may encounter· another breakpoint-which could have its own command list, leading to ambiguities about which list to execute.
If you use the 'silent' command in the command list, the usual message about stopping at a breakpoint is not printed. This may be desirable for breakpoints that are to print a specific message and then continue. If none of the other commands print anything, you see no sign that the breakpoint was reached.
New in version 2.5.
s(tep)
Execute the current line, stop at the first possible occasion (either in a function that is called or on the next line in the current function).
n(ext)
Continue execution until the next line in the current function is reached or it returns. (The difference between "next" and "step" is that "step" stops inside a called function, while "next" executes called functions at (nearly) full speed, only stopping at the next line in the current function.)
r(eturn)
Continue execution until the current function returns.
c(ont(inue))
Continue execution, only stop when a breakpoint is encountered.
j(ump) lineno
Set the next line that will be executed. Only available in the bottom-most frame. This lets you jump back and execute code again, or jump forward to skip code that you don't want to run.
It should be noted that not all jumps are allowed -- for instance it is not possible to jump into the middle of a for loop or out of a finally clause.
l(ist) [first[, last]]
List source code for the current file. Without arguments, list 11 lines around the current line or continue the previous listing. With one argument, list 11 lines around at that line. With two arguments, list the given range; if the second argument is less than the first, it is interpreted as a count.
a(rgs)
Print the argument list of the current function.
p expression
Evaluate the expression in the current context and print its value. Note: "print" can also be used, but is not a debugger command -- this executes the Python print statement.
pp expression
Like the "p" command, except the value of the expression is pretty-printed using the pprint module.
alias [name [command]]
Creates an alias called name that executes command. The command must not be enclosed in quotes. Replaceable parameters can be indicated by "%1", "%2", and so on, while "%*" is replaced by all the parameters. If no command is given, the current alias for name is shown. If no arguments are given, all aliases are listed.
Aliases may be nested and can contain anything that can be legally typed at the pdb prompt. Note that internal pdb commands can be overridden by aliases. Such a command is then hidden until the alias is removed. Aliasing is recursively applied to the first word of the command line; all other words in the line are left alone.
As an example, here are two useful aliases (especially when placed in the .pdbrc file):
#Print instance variables (usage "pi classInst")alias pi for k in %1.__dict__.keys(): print "%1.",k,"=",%1.__dict__[k]#Print instance variables in selfalias ps pi self
unalias name
Deletes the specified alias.
statement
Execute the (one-line) statement in the context of the current stack frame. The exclamation point can be omitted unless the first word of the statement resembles a debugger command. To set a global variable, you can prefix the assignment command with a "global" command on the same line, e.g.:
(Pdb) global list_options; list_options = ['-l'](Pdb)
q(uit)
Quit from the debugger. The program being executed is aborted.
Subscribe to:
Posts (Atom)
