Testing is a major part of any project lifecycle, whichever methodology is chosen to carry a project forward.† What is often not fully appreciated by people outside of testing is that the test process has a lifecycle too.
Below is the Automated Test Life-cycle Methodology (ATLM).† This incorporates the added stages applying to the decision to automate testing, acquisition of test tools, and the introduction of the tools to the project.
Figure 1.1: The Automated Test Life-cycle Methodology (ATLM).
Although this essay shares itís name with stage three of this methodology, it is difficult to talk about stage three without first addressing the reasons for choosing to automate tests and the steps that need to be taken during acquisition of tools.
From here on Automated Software Testing will be abbreviated to AST for simplicity.
AST tools are good at simulating a large variety of input data to test how a program copes with irregular and unexpected data.† They can also do this quickly and with a level of consistency that a human could never achieve.† From a quality perspective AST is ideal, as it will not make errors inputting a test case even if it has been repeatedly inserting data for the last few weeks.
If humans were to do the same job as the AST tools their performance would be error prone and it would be a labour intensive task.† Doing this kind of repetitious work can be mundane and incredibly disheartening and usually leads to low team morale and a high turnover of testing staff as they get bored and move on.† This lack of consistency is not good for a project.
Using AST correctly still takes longer in the first instance than manual tests.
Table 1.1: Automated versus manual testing Ė GUI testing experiment results
(Source: Galin, 2003, p.244)
The above table shows that for a single testing schedule manual tests are far more time and cost efficient.† When regression testing (re-testing) is needed the benefits of AST shine through.† When reusable test scripts have been produced there is no need for any of the preparation work to be re-done, the only cost is running time, which is considerably shorter than in manual testing.† On a project that needs multiple sets of regression tests AST could be a very practical choice.
AST tools are also used to put software under a heavy load and test it under data rate and processor loading conditions that it would not usually face.
Examples of where AST tools are employed successfully are:
∑ Banking systems Ė Need rigorous testing with all possible data under all loading conditions as any malfunction could cost the bank substantially.
∑ Real-time control systems Ė Although the range of inputs may be limited there is a lot at stake if a control fails (e.g. nuclear power station).†
AST tools are not the holly grail of the testing world, they are merely an alternative to manual testing.† If they are applied to the correct situation, at the correct time, with the correct amount of support and planning they can yield fruitful results that can save a project time and money.† If great care isnít taken with decisions of tool suitability, integration and implementation they can be a major thorn in the projectís side, leading to product delivery delays and worse.
The software industry is under pressure from customers (and even government) to streamline the software product acquisition process.† One area of the software engineering lifecycle that requires lots of time and resources is the testing phase, making it an obvious target for improvement.† Introducing AST correctly can help to achieve this goal.
AST has often failed to live up to expectations, usually because expectations were too high and implementation was carried out in less than perfect conditions.† Introduction of AST takes a lot of resources and projects underestimating this has blemished many peoples view of AST through no fault of its own.
Thereís an old saying that goes ďWhen you first learn to use a hammer everything looks like a nailĒ.† Just as everything isnít a nail and canít be used with a hammer, not all required testing lends itself to AST.
Introduction of AST will probably mean a re-evaluation, re-organisation and retraining of the test team as their roles and responsibilities will be different to when manual testing is being carried out.† During the decision to implement AST there must be an element of AST experience on the test team, whether that be a permanent test team member with the necessary experience or another qualified person who is drafted in to oversee the process.
A tool may have great potential, but if the users donít take advantage of its attributes it might be no use at all.† With a fast learning curve nothing counts more than experience and having a seasoned veteran of AST on the team (even in the short term as a mentor) will be invaluable.
Prior to AST introduction there needs to be a review of the project schedule to see how it will be affected.† If the project has already started then it may turn out to be counterproductive to use AST as there wont be enough time to progress the testing life cycle to a point where testing will be available and ready to start at the time it is required.
Although a test tool may have been successful in prior projects it doesnít guarantee that it will deliver the same results on the project concerned.† It is important that a test tool is evaluated for itís possible performance on this project by making sure that it will help meet the projectís system requirements and function in the required environment.
Acquisition of a testing tool should be carried out after researching available options and evaluating their features against the projectís testing needs.† Ideally, endorsements of the tool should be checked to validate the toolís marketing claims prior to purchase.
Acquisition of a testing tool should be followed up by prompt testing to ascertain whether the product specification holds true, and whether the tool will actually perform the job that it was purchased for.† Test tools run alongside the program that is being tested.† This can cause unforeseen problems if the program under test is already stretching the environmentís resources and test results suffer as a result of the testing process.† Analysis of whether the programs are compatible with each other and the common resources that hey will use must be undertaken at the earliest possible stage, giving the test team time to solve any problems that are found prior to tool introduction.
Any updates for the testing software should also be tested to make sure that it still meets the projectís requirements.† Just because the update has been tested and found to still test other projects adequately it doesnít mean that the same applies to every project instance.† This should take place in a controlled environment (as for all testing) where it can be confirmed beyond question that the product will still perform the necessary tests to the same standard that the original accepted version did.† If this isnít done and later it is found that errors testing software caused project errors to be left undetected the whole post-update testing schedule could be invalid.
Itís important that the introduction of AST is not seen as an event that just happens and can then be ignored. AST introduction is a process, with certain conditions that need to be satisfied before it can even start.
Historically, AST has been implemented using ill equipped test teams that have not used the tools effectively.† The major criticism being the inability to create test scripts that are reusable enough.† Although the test scripts have worked for the current version of the software, they were useless on subsequent releases even if the new version incorporated only minor changes. Since creating the test script undoubtedly required more effort than a manual test of the same type, this is an unacceptably inefficient usage of man-hours that could have lead to an overrun of the testing schedule and ultimately a delay to the product delivery date.
When this has happened the AST has usually been unduly criticized when the fault lay in the implementation of the tool and not the tool itself.† As a result, the people involved have lost confidence in AST and subsequent attempts to use these tools have been rejected due to the bad experiences.
Prior to introduction, the current test process needs to be analysed so that it can be evaluated against the following list of prerequisites.† Only when the list has been successfully completed can implementation of AST be seriously considered.
∑ Testing goals and objectives have been defined.
∑ Testing strategies have been defined.
∑ Tools needed are available to implement planned strategies.
∑ A testing methodology has been defined.
∑ The testing process is communicated and documented.
∑ The testing process is being measured.
∑ The testing process implementation is audited.
∑ Users are involved throughout the test program.
∑ Test team is involved from the beginning of the system development lifecycle.
∑ Testing is conducted in parallel to the system development lifecycle.
∑ Schedule allows for process implementation.
∑ Budget allows for process implementation.
∑ Understand whether the organization is seeking to comply with industry quality and process maturity guidelines (i.e. CMM, ISO).
If the current test process is out-of-date or inadequate in some way, this is an opportunity to review the validity of it and consider the introduction of the Automated Test Life-cycle Methodology (ATLM), which will complement the AST perfectly.
The projectís system requirements need to be reviewed and a detailed plan of what the test tool will be used for drawn up.† The testing environment also needs to be defined and confirmed before it can be verified that the AST tool will be of any use.† Ideally, a project prototype and system overview would be available at an early stage of the project so that the test team can progress their testing life cycle and start designing and developing test scripts as soon as possible.
It has been pointed out before that people have preconceived notions about AST.† Some are positive in that they expect tools to do far more than they actually can and some are negative due to past bad experiences.† Some people may not know much about AST but fear that its introduction will cause human testers to become surplus to requirement, so will naturally be resistant.
It is a good idea to manage the expectations of the whole project team and keep them informed about what is going on.† A demonstration of the tool is a good way to communicate the effects of the toolís introduction to the project to the people that will be and think they will be affected by it.
∑ Introduction of AST to a project is a process, not an event.
∑ In order to ensure its success the tool should be introduced at the start of a project, so that the test team can utilise it fully by the time testing is expected to begin.
∑ Make sure that the tool is being used for the right reasons and that it is the right tool for the job, not just the one thatís been used in the past.
∑ Test the test tool prior to its introduction.† Finding out that itís not adequate halfway through a project would be devastating.
∑ Allow the resources to write re-useable test scripts.† Otherwise manual testing is proffered.
∑ Manage peopleís expectations of the tool.† If people expect too much they will always be disappointed no matter what the outcome.
††††††† From an article appearing in Methods & Tools, Summer 1999 (pp. 11Ė15) by Elfriede Dustin, Jeff Rashka and John Paul.
††††††† Software Testing: Testing Across the Entire Software Development Life Cycle
by Everett, Gerald D., D., McLeod, Raymond Jr. (ISBN 978-0471793717)
††††††† Software Quality Assurance: From Theory to Implementation by Daniel Galin, (ISBN 978-0201709452)
††††††† Automated Software Testing by Elfriede Dustin, Jeff Rashka and John Paul (sections of chapter 4), ISBN 0-20-43287.