Extending Our Coverage
Automated GUI test scripts are by far the most fragile and expensive to maintain, although WebTest’s features minimize the need for changes. These scripts were fine for smoke tests, to make sure nothing major or obvious in the application was broken, but we didn’t want to do detailed functional testing this way. Also, we still needed a tool to support our plan to drive development with customer-facing, executable tests and examples.
Once we had traction both at the unit test and GUI test level and could take a breath, we looked for a tool that filled up the big gap in the middle. We looked at FIT and FitNesse (which is essentially FIT using a wiki for the IDE) since they allow a non-programming user to write test cases in a tabular format and programmers to easily write fixtures to automate them. These tools basically replace the UI. They allow you to send test inputs to the code, operate on them, return actual results and compare them automatically with expected results. The results turn red, green or yellow, and we love color coding. Both tools have a large user base and active mailing lists. We liked FitNesse’s wiki component, so we decided to try it.
FitNesse turned out to suit our needs for documenting not only the features we developed, but other information about maintaining the application. We found that it was easy to learn how to define test cases and write the fixtures to automate them. Integrating the test suites into our build process took longer, but was doable. It was easy to write both high level tests to give the big picture, and executable tests to capture the detailed requirements.
Sometimes you get unexpected benefits from tools. We understood that creating FitNesse tests would require work from both a tester, to specify test cases, and a programmer, to automate them. We found that this enforced collaboration enhanced communication within the team. The resulting communication flushed out misunderstandings and wrong assumptions early in the development process. Test tools aren’t just for automation!
Be Open to Experimentation
Several months later, we had good coverage on the GUI side from our smoke test scripts, a safety net of unit tests for new code (plus the benefits of TDD), and used FitNesse tests to help guide development. We still had unfulfilled testing needs.
One difficulty was setting up test data. We had test databases, but creating data to achieve a specific scenario was often a tedious manual task. When we hired a tester with extensive Perl experience who was also interested in learning Ruby, we decided to give WATIR a try. Although we already had a GUI test tool, our new WATIR scripts allowed us to quickly create any combination of data we needed, making it much easier to do serious exploratory testing. No doubt, we could have probably done the same thing with the tools we already had, if we had spent the time researching how to do it. Having someone with skills to get us all up to speed with WATIR scripts made that the path of least resistance, although we had to plan time to learn Ruby. (I enjoyed learning Ruby, and enjoyment is an important factor too!)
Another item missing from our toolbox was a load test tool. When we needed a way to do load testing a couple of years down the road, we started doing some research. I asked the members of the agile-testing Yahoogroup for load test tool suggestions. Recommendations included Httperf, Autobench, OpenWebLoad, Apache Bench, Apache JMeter and The Grinder. After considering factors such as scripting language, learning curve, result content and formatting, recommendations from other users, and compatibility with our application, we budgeted time for trial runs of both JMeter and The Grinder. We had the best results with JMeter, a Java desktop application, and found that a tool called Badboy helped us create the JMeter scripts. We liked the statistics it generated, and how they were displayed. We were productive with JMeter quickly and were happy with our choice. As with all our tools, though, we’re always open to new developments on the tool front.
Even after you have test automation in place, there is so much development of new and existing tools that it pays to keep up with what’s new. Monitor mailing lists such as http://groups.yahoo.com/group/agile-testing, attend local user group meetings, read online and print publications (such as this one!) and attend conferences when possible to catch the latest tool buzz. Recently a couple of our team members saw demos of Selenium, and the agile-testing mailing list had posts from happy Selenium users. It has some potential advantages for us, so when we can budget time for it, we’ll check it out. We don’t intend to throw any existing tools away, since they’re still working for us. But if there’s a better way to automate particular types of tests, we’re open to trying it.
Your needs might change and evolve, as ours did. As team members come and go, your team’s skill set might change. When our WATIR expert move on, my Java programmer teammates bought Ruby books and started learning more about it. However, now there is a Java-based option, WATIJ, that might be more appropriate to our needs (there is also a .NET version, WATIN). Part of Selenium’s appeal is the ability to write the scripts in Java (which the programmers like) or Ruby (which I like). More and more tools, especially the open source ones, support multiple languages and provide more flexibility.
Tools aren’t just for automating regression tests or helping with exploratory testing, either. Scripting languages such as Ruby can automate all kinds of tedious tasks that teams need to do. Get ideas from books such as _Everyday Scripting with Ruby_ by Brian Marick.
Sometimes tools can be combined for even more advantages. Groovy scripts can be integrated into WebTest scripts to provide more flexibility. Selenium tests can be run from FitNesse. Those are just a couple of examples. Don’t limit your thinking to an individual tool’s features.