Day 1 of SPLC 2006 also saw me at John McGregor’s Creating resuable test assets in a Software Product Line session.
John wears various hats including Conference Chair for SPLC 2006, Associate Professor at Clemson University and visiting Scientist at SEI, as well as being an independent consultant. John has a lot of practical experience with organisations planning for or using a Software Product Line approach so I was looking forward to what he had to say on the difficult issue of Software Product Line testing.
I didn’t count, but I guess around 20 people were present at this session, many of them involved in testing in some respect or other in Product Line or aspiring Product Line organisations.
The session explored (scalable) techniques for reducing test effort, while maintaining or improving quality, in a Product Line setting, through reusable assets. John also advocates testing across the whole life cycle from inception through to deployment and evolution, correctly in my view, and this naturally leads to testing as early as possible. The result was that a lot of ground was covered. Some of the main observations:
- Automated testing is a critical success factor. (Is anyone in the industry not convinced about this now?)
- Align your test strategy to the Product Line strategy. For example, if speed is an important Product Line goal, then you should be looking to build your automated testing capability further.
- The testing group, like the rest of the organisation, will need to change to adapt from testing a set of independently-developed products to a set of co-developed products.
- Test assets can be defined at various levels of abstraction corresponding to the levels of design abstraction in the software.
- The vast majority of defects can and should be identified in Unit Testing.
- If exhaustive testing isn’t possible – and it generally won’t be in a large Product Line - then sampling techniques can be used to reduce risk.
- John noted a technique known as Orthogonal Array Testing, or pairwise testing, that can be used to dramatically reduce the number of asset configurations that need be tested. This technique focusses on testing two-way interactions. The technique works because most faults (>90%) are typically caused either within a single asset or by some two-way interaction between assets. There’s a good article on pairwise testing here if you wish to explore this area further.
- As your software evolves, so must your tests.
John also noted a number of common anti-patterns in Product Line development:
- A lack of management support will lead to the failure of a technically-driven reuse effort. (Other people, or pit-diggers as I once heard Jim Dager call them, can also sabotage such efforts, but will be less inclined to do so if management are on board.)
- Not involving people from QA in the Product Line formation. They should be involved to help prepare the business case and the Product Line Scope.
- Reducing the time taken on requirements specification can reduce testability. This is often compounded by a general weakness in requirements reviews.
- Test assets not being managed as core, reusable assets, in the way that core, reusable software itself is managed in a Product Line. This is odd as the Product Line Architecture can (often) be leveraged to create the Test Architecture and in some cases it may be possible to use the same variability mechanism in the test asset as you use in the corresponding software asset. Test plans and other artefacts can (or should) also be reusable.
I’d have no problem recommending John’s tutorial to anyone interested in Product Line testing. Maybe he’ll run it again at SPLC 2007.
A full report, including this tutorial report, will appear on the Software Acumen website shortly.
