Manual testing is a key element to a hardened QA process allowing for the human mind to be used in tasks that lend themselves to critical thinking or are simply not automatable such as:
- Exploratory (also called ad-hoc) which allows testers to test strange and unusual codepaths
- Use Case which intends to simulate as closely as possible the end to end solution that will be deployed
- Usablility which aims to test whether the product meets the timing, performance, and accessibility needs of the end user
The only way to have enough time to do these kinds of tests (my favorite is exploratory) is to automate as much of the functional and regression testing as you can. Once you know a proper and reliable procedure for producing a functional test it is time to stick that in a regression suite so that it no longer needs to be run by a human.
At Eucalyptus, we have a great system for provisioning and testing a Eucalyptus install (developed by Kyo Lee) which is currently in its third iteration. As new features are added and as the codebase expands, we have identified the need to tie this system into others in order to make a more fluid transition of our manual tests into the automation system. Currently we use three main systems for evaluating the quality of Eucalyptus:
- Jenkins to continuously build the software
- Testlink (an open source test plan management tool) to maintain our manual tests
- Automated provisioning system to run regression suites against the code base.
- Upload your test script (hopefully written with Eutester) to a git repository.
- Create a testcase in Testlink that defines the procedure and any necessary preconditions. If the case to be automated is already in manual testing, this can be skipped.
- Add info to the testcase that describes where your new test script lives.
- Add the test case to a testplan through Testlink.
- If the test was added to the continuous integration suites, it will automatically run against all of our platforms
- If the test was added to a personal testplan, it can be kicked off manually using the Jenkins UI
With this flow defined lets take a look at how the individual pieces will interface.
Jenkins to Testlink:
Testlink is a flexible tool that allows testplans with great complexity to be organized efficiently. Although this on its own makes Testlink my top choice for multi-platform manual test organization, the real secret sauce comes from the API that it presents. The API gives you programatic access to the metadata of any testplan by using an XML-RPC interface. In order to access this API through Jenkins, I could hack together a set of Python scripts to speak Testlink but instead all that work has been done for me in the Testlink Plugin. This plugin allows me (as part of a Jenkins “build”) to pull in a testplan with all of its testcases and their respective custom fields and execute a build step/script for each case. In order to take advantage of this I have added a few crucial custom fields to each test case in Testlink:
- Git Repository
- Test Script Path
- Test Script
- Script Arguments
In the above build step I am simply cloning the repo that the testcase exists in, then running the script with the prescribed arguments. This gives the test case creator enough extensibility to create complex testcases while standardizing the procedure of submitting a test case to the system. Another advantage of including a configurable arguments field is that the same script can be parameterized to execute many test cases.
Jenkins to QA Provisioning
Our current QA system has recently been upgraded to include a user facing API for deploying Eucalyptus against pre-defined topologies, network modes, and distributions. In order to hook into this system we will be using Python and the json library to parse the data returned from the QA system. The types of calls we will be issuing are:
- Start provisioning
- Wait for provisioning to complete
- Free machines used in test
In this blog post we have shown how this integration can be used to quickly move a test from being manually tested to being automated but in order for this integration to be a true success it must deliver on at least these other use cases which will be described in further posts:
- Continuous Integration
- Test Report Generation
- Developer QA system sandbox