An Agile Tool Selection Strategy
The whole team approach means asking ourselves, “What skills do we have on our team?” Do any team members have extensive experience with particular test tools or types of test tools? What programming and scripting language competencies exist on the team? How much technical expertise do the testers have? How about the business people who might be reviewing or even helping to write tests?
What types of tests are you automating? Unit, integration, functional, security, or do you need to do performance or load testing? How robust do your test scripts need to be, and how much can you spend on maintenance? Are you planning to do data-driven or action keyword type tests where the tests accept a variety of input parameters and have a lot of flexibility? Or are you looking for straightforward, low-maintenance tests? Can you test at a layer below the user interface, or do you have an architecture that makes that difficult? These are all considerations when shopping for a test tool.
With a variety of test needs, consider that you may need a variety of tools. We tried to keep an open mind on what might solve a particular automation problem, and we were willing to experiment. We’d pick a tool to try for a few iterations and see how we liked it. Getting up to speed on tools to the point where you can effectively evaluate them takes time, so be sure to budget plenty of time in your planning.
Our Tool Selection Process
If your team has specific needs that may not easily be met by a generic test tool, and you have the skill set to accomplish it, consider “growing your own”. This way, you get a tool is customized to your needs and integrates well with your application. Many teams have chosen this route, which is why there are so many excellent open source tools available today. If you ‘home brew’, you still need to consider your test tool requirements. For example, if non-programmers need to specify tests, you’ll have to develop an interface to allow that.
In our case, our financial web-based application didn’t seem all that unique, and we were a tiny team without a lot of bandwidth for tool development. Our management included funds for test tools in the budget, so we looked for an outside solution.
To illustrate how your tool selection process might work, come back in time with me to late 2003 / early 2004 when we started our search. If we started from scratch in 2007, there would be even more options to consider! A couple of great places to start your search for web testing tools are www.softwareqatest.com/qatweb1.html and www.testingfaqs.org, which list both commercial and open source tools, and www.opensourcetesting.org, which provides information about all kinds of open source software testing tools. Another good resource are members of your local testing/QA user group, and testing-related mailing lists such as http://groups.yahoo.com/group/agile-testing.
We had to address our various needs with tools. Unit testing was a no-brainer. With a Java application, JUnit is the obvious choice. The team got started writing unit tests right away, as they would form the foundation of our regression tests. Next on the priority list was GUI testing.