Top challenges faced by Product/Test Owners during Product Testing

Performing software testing looks always to be an easy task for an outsider who does not belong to testing community but for most of the product/test owners it has always been a challenge to do it RIGHT most of the time. There are various instances of defect leakages in the past that have caused multi-million dollar losses to various companies and agencies. For a more detailed list of such issues you can visit site (http://blogs.zdnet.com/projectfailures/?p=427)

The product/test owners who are part software PDLC often face the following challenges that are either addressed by them internally or they need to hire external consultants to overcome them.

#1 Are we doing the right thing the right way?

#2 Is my testing team doing enough testing?

#3 Wish I had more budget to test more…

#4 The product has to ship tomorrow not sure if will have tested it enough

#5 Is my product of acceptable quality?

#6 Shall I outsource?

With all the above challenges, the idea is to understand whether testing is taking the correct route and leading to a greater understanding of the product, its strengths, weaknesses and market potential.

The first question that comes to mind of a product/test owner, whenever any testing assignment is started is “What to test”. It is quite a tedious task and needs a thoughtful thinking while deciding on the areas that needs to be tested for delivering a better quality product. If we do not define it at the starting, it becomes a very painful task going forward and various stakeholders keep on guessing the outcome of testing.

Once it is decided as to what to test, the next challenge comes in “How to test” It is indeed very important to select the right tools and activities amongst the that should be performed to do the right kind of testing to achieve desired test coverage. Desired test coverage! Did I talk about this?

This is again another challenge to decide on “how much to test” so that the test team can say that the product is ready to ship. It is tough to clearly differentiate and define the exact time of closure of testing activities. Once the scope and methodology is selected and decided upon, the next is the identification of Risks that needs to be identified upon and a mitigation plan needs to be formed for the same. Risk identification should include risks throughout the life cycle in the areas like cost, schedule, technical, staffing, external dependencies and maintainability and should include organizational and programmatic political risks. Risk identification need to be updated regularly. Risk mitigation activities need to be included and communicated to various stakeholders.

Even if one has clarified and answered each of the questions that are discussed above, one doubt still remains throughout the testing process. It is “Will all the bugs be reported during testing”. Though everyone knows that Testing does not guarantee reporting of all the bugs, but effective and efficient testing does guarantee the reporting of all the critical defects that exist in any product for its smooth running after it goes live. Still, the product/test owner has to face a lot of heat if even a single defect is found by end users after going live and there is a question mark that is put on the complete testing cycle. So it becomes a big challenge to find out the bugs during the testing cycle.

Product/test owner has to determine at the outset whether their teams are doing enough testing. This is not as simple to identify as most of the time product/test owners do not have complete visibility into the testing process that is being followed by the testing teams. There is sometimes no way to identify whether the team is spending the time correctly on various tasks that are being assigned to them. To identify that the test team efforts are being put in the right direction for getting adequate test coverage is a big challenge. If these efforts are not measured and corrected at the right time, then it might create a bigger problem at later stages and the overall test coverage achieved might not be up to the expectations of the stakeholders. And after this, to validate the product status that has been reported by the team is also a challenge that is hard to meet.

At the same time, product/test owner have to examine whether the product is finally market ready and of acceptable quality for clients.

One of the major challenges faced by product/test owners is to identify the correct budget for testing activities. It is often seen that there is not much introspection done to decide on the testing budgets and they are fixed more on the random basis e.g. fix testing budget as 30% of development budget or so on. But for a product/test owner it is tough to plan (as much as possible) for the budget that will be required during testing of the product? The correct identification of the correct tools, resources, skills, required during testing, remains a major bottleneck for the identification of the correct budget.

Finally, they are also grappling with the question related to outsourcing and considering outsourcing as an option when drawing up testing strategies. But for the beginners, thinking outsourcing has its own challenges like how can I know who is doing what within the team, does the team understand the product domain and functionalities completely so as to take the testing assignment and whether they value my IP and will it remain in safe hands even if I decide to outsource.

In my next post, I will be detailing out on some of the possible solutions that if implemented within any testing team, can help in overcoming the challenges listed above.

Do feel free to put in your comments on this post.