Maximise test coverage by writing effective Test Cases

Once your Test Plan has been structured and completed with all the relevant and consolidated information, based on the requirements gathering and specification process, the real creative work can begin, i.e. thinking about and designing the Test Cases that will be executed on the software system under examination

.

Although this phase might seem very boring, especially when it is not undertaken with the right spirit, it represents one of the most important of the test process, as only through a proper Test Case design is it  possible to achieve maximum test coverage.

There are of course several approaches and techniques, but before describing those that have provided me with a high return on investment, I would like to explore briefly the most used by the testing community. Obviously we need to distinguish between the different scenarios like a new project, a redesign of an existing one and last but not least a project under maintenance which can be subject to bug-fixing and/or small enhancements. 

In the case of new projects, life should be straightforward, as in theory there should be enough time to do everything well. However this rarely happens  because when there is too much time available for thinking, people tend to spend it on planning and discussing what might be the right approach, which test types and levels have to be forecasted. They then realise  too late that what was planned cannot be put in practise due to a shortage of time. 

However, if the available time is used properly,  a proactive approach is usually the best one because it is possible to concentrate on the test design effort before any code has been written. Then refining the test cases as soon as the first version of the software is ready for testing (even though I strongly recommended getting access to review the implementation during development) and at the same time giving the developers the chance to verify that they have taken into account all the conditions stated for each test case together with the expected output, will result in a higher quality release that will be put through formal testing at the end of the development phase. 

Obviously this approach is not only proactive but also iterative, as it is developed over the project life cycle rather than either upstream or downstream of the coding phase; but of course the availability and cooperation of the development team is vital.

In such a case the Test Design techniques I would suggest, also to prepare test driven scenarios, are the Equivalent Class Partitioning together with Boundary Value Analysis and a further refinement using Decision Tables. In this way the Test Cases will ensure most conditions are covered, but this is still not enough, as the objective is to cover all the business processes and the possible scenarios as well. To do that a good approach would be to use Mind Maps, which allows us to draw what test/business analysts have in their heads, and since it is usually based both on the experience in the testing field (which is very important) and on the business domain (having correctly understood all the functionalities and their interactions) it gives us a clear picture of the system behaviour.

An additional benefit is provided through the ease of translating such a representation into the Test Management platform, as the structure of the Test Scenarios is already prepared – The Mind Map –.

The contrary approach, i.e. a reactive one,  would normally be adopted for projects under maintenance. This is because on the one hand it is not possible to determine when and where software failures will occur, on the other hand for small enhancements (which does not merit a new project) there is not usually enough time available to follow a proper test process. In this case the existing test cases are typically updated to take the changes into account.

A more complex sub-scenario must be faced when no Test Cases are available for the existing functionalities (which is very common for projects developed when the organisation was not yet well organised from the testing perspective), one which requires a very fast design process in order to react quickly to changes.

In such cases the best approach is to start creating the specific test cases to test the functionality/feature subject to changes and incrementally add cases every time new defects/small enhancements occur. But of course this requires more resources as the resources that have already been allocated will be busy with test execution.

To summarise, for existing projects the test design is mostly based on the experience of the test personnel involved and will usually require additional testing resources.

For redesign projects the situation is slightly different, because even though the existing Test Cases, if any, can be properly updated as the project plan should take into account the whole Test Process, forecasting the right time for Test Case Design and validation, can become more complex if the system behaviour is likely to differ heavily from the existing one. In this case the approach mentioned above for band new projects can provide some help.

If this is not the case the advantage is represented by the possibility of doing Exploratory Testing as the actual implementation is available both for investigation and to get familiar with the processes; moreover the key users feedback and suggestions can be exploited to better design effective Test Cases.

I have briefly explored the most common approaches, now I would like to describe the approach I typically apply and that for recent projects has allowed me to maximize the return on investment, a topic that is very important for organisations.

I have not reinvented the wheel, as it is a combination of the approaches mentioned previously.

Basically I always follow an iterative approach, no matter whether or not I am working for a brand-new project or for an existing one. My Test Strategy and the consequent Test Design approach, is essentially the same with slight adjustments in order to fit with the organisation standards, methodologies and techniques.

My first requirement is to have access to the entire set of project documentation, both business and technical.  Then, as soon as I can get all the answers/explanations I need, I organise some sessions with key users and specifically:

• For a new project I meet directly with them in order to understand their vision (especially from the usability perspective) and how they would like to interact with the new application, which functionalities represent the core and which are desired but very important to maximise their effectiveness in performing operational duties
• For an existing project redesign/under maintenance it is even better as I am typically in a position to observe how they work (basically doing exploratory testing together) and to understand if the application is forcing users in to unnatural usage patterns and/or if there is space for improvement. I usually spend a lot time with them to be able to quickly report back to the project manager and verify if the feedback/learnings can be considered

Spending time with the key users/subject matter experts/final users will bring you many benefits that are hard to imagine. Moreover they really appreciate this kind of involvement/contribution and this makes them more willing  and open to changes, a subject to which there is always a strong resistance.

Where a project has a prototype or set of navigable wireframes that is available before the implementation commences  (i.e. during the functional analysis phase) it is a good practice to organise some Usability Testing sessions, as doing so makes it possible to anticipate problems which if missed could result in expensive rework if the key users are only able to look at the application immediately after development has been completed (this is true even if an Agile approach is being used, as also in case of intermediate validations – UATs – carried out at the end of each sprint, if the key users do not have the possibility to see a prototype before development it could result in changes being needed in order to provide the desired behaviour).

When the working session and the Usability Testing are performed together with the key users, it allows the Test Cases to be better organises and designed. This is because by focussing on the functionality and interaction and how the final users will use them, allows us to prepare business oriented Test Cases and to think of all the possible related cases in advance.

Once the skeleton of the Test Cases has been prepared, it is now time to apply the techniques mentioned above, i.e. Equivalent Class Partitioning with Boundary Value Analysis and Decision Tables (there are many other techniques but these three are the techniques I prefer and those that can be automatically implemented once experience has been acquired).

At this stage things are almost complete but our Test Cases need further refinement; how can we do that? Easy, we need to push developers to show the units, modules and user stories that have been implemented. This is an activity that needs to be carried out over the entire length of the project, by doing this we do not need to wait until the end of the development/sprint to see the actual implementation with our own eyes and we refine the Test Cases incrementally during the project life cycle. An additional benefit of this approach is having the possibility to anticipate the Test Execution as well, removing the main issues that could result in a costly rework if undertaken downstream of the development phase.   

Last but not least, by following such approach it is possible, both in case of manual and automated testing, to estimate the Test Execution time and to add it as a reference for each Test Case. This provides valuable information for the Project and Test Managers, who can easily adjust their plans based on the Test Execution estimation.

To summarise, the approach I suggest requires a high level of proactivity and interaction/collaboration  with the key users but this will definitely help you in achieving the desired result, i.e. the highest quality for your projects. 


Massimo Forno - IxmaSoft CEO

Software Testing: adapt and grow with the Cloud

Cloud Computing has certainly brought a lot of new possibilities in terms of business and clearly, as the arrival of a new technology or methodology often requires, Software Testing needs to adapt and grow in this direction.

Several new testing services are already available and others will follow shortly. This helps us to understand who has benefited the most from this type of computing, i.e. the software testing companies which aside from the value proposition offered by these new services, they can now reduce the Cost of Testing by taking advantage of multi-tenant and off-premise architecture. This helps them to avoid the purchase of expensive hardware and software and at the same time allows them to scale faster and less expensively.

Moreover configuration of the test environments has become easier and almost instant, allowing for faster test execution which has the potential to increase the number of test sessions that can be run in the same time period.

As mentioned earlier companies, whose core business is the provision of testing services, can exploit this new architecture in ways that result in clear benefits for software vendors; in fact their goals often coincide since their main concerns are typically related to costs and budget. 

On one side, software houses tend to invest more in business analysis and development activities, asking test teams to be both effective and efficient with limited resources. On the other side, testing providers want to maximize their revenues by trying to reduce costs; this is especially true in those countries where less effort is invested in testing due to psychological and economical barriers resulting in compulsory cost savings policies.

This article examines how Cloud-based testing is positively affecting several types of testing.

Functional Testing is probably the least affected type of testing, except in the case of test outsourcing when testing companies are asked to take charge of non-web based applications, which require direct access. 

This situation is a little more complicated because, even if the issue of direct access can be resolved by using Remote Desktop Services to access the software house test environment, or by allowing the testing company to directly deploy the application in their internal test environment, software vendors are not always willing to grant access to their network to third parties and/or share the application code.

In this case the Cloud can help a lot by providing quicker and safer access to the application subject to test.

Non-Functional Testing is more complicated and certainly the most affected, firstly because of the different test types (Performance, Security, Compatibility, Usability…) and secondly because it usually requires huge investment in hardware and software licences.

Performance Testing (including Load and Stress Testing) probably gets the biggest benefit from Cloud computing, considering both the economic aspects and the needs of scalability in order to gradually increase the simulated load.

Using the Cloud means paying only for what is needed and only for the required time. This means that it is no longer necessary to purchase expensive annual licenses for two or three months of testing per year. Server provisioning in the Cloud is almost immediate and decommissioning means that there are no problems associated with reallocation. Furthermore the test environment can be shared with the development teams, which inevitably helps to improve the communication between testers and developers.

With this configuration, testing companies can maximize their revenue because their Cost of Testing decreases proportionally considering that they are no longer obliged to buy expensive hardware and software, ensure business continuity or manage frozen hardware and software requirements.

On the other hand, software vendors now have a bigger range of options as many testing providers have already moved their services to the Cloud often offering Test Agents and Controllers ready to run homemade scripts. This still means that software houses can decide which logic and functionalities to test without having to worry about infrastructure costs and configuration; this allows them to use on-demand testing tools that would otherwise be very expensive.

Security Testing executed in the Cloud provides significant benefits, firstly because many aspects of this type of testing are computing and storage intensive and secondly because Cloud computing can be considered as the natural environment where both periodic and occasional activities can be executed. This allows accurate and actionable results to be obtained whilst paying just for what is needed for the required time.

Moreover as this is managed by IT specialists, Cloud usage is usually safer and more efficient when compared to traditional internally managed test environments.

Compatibility Testing is another type of testing that demonstrates how this architecture can reduce costs by avoiding the need to buy licences for multiple operating systems (and different versions of those operating systems).

Cloud based testing can also help to speed-up the creation of multiple operating system and browser combinations, even when they are managed using virtualization techniques.

Conclusion

From the software solution provider’s perspective, independent of whether or not a company has its own Software Quality Control Unit (and / or test teams), there are clear economic and performance advantages to using Cloud-based testing.

Testing provider companies are taking increasing advantage of Cloud usage, especially for those services that require a big infrastructure in order to perform test activities, particularly for non-functional testing.

To summarize the major benefits:

• Cost savings both for software vendors and testing providers
• Easier and faster test environment setup
• Easier, faster and cheaper resizing of the test environment
• Faster test execution which allows more test sessions to be executed
• Increased quality of the results due to the IT specialists managing the Cloud

Massimo Forno - IxmaSoft CEO - Articolo pubblicato nel n°20 di Testing Experience - The Magazine for Professional Testers, Dicembre 2012