As we know, “static analysis” is valuable and important in developing work. Many genious tools like
jshint help the developer to find the problems earlier.
Test Automation is another style of developing work, but unfortunately we always ignore its quality. The result is, in most scenarios, the Test Automation asserts are hard to maintained and unefficiency.
I wrote a simple static analysis script for RobotFramework during the Sprint Festival, tried to add the lost part of
Test Automation. In this blog, I want to discuss how I think and implement.
As the cases of RobotFramework may be saved as a HTML, a txt or csv file, we care about the logical structure instead of the plain text. So in the first step, I decided to rely on
robot.api, and listed the checkpoints as below:
- show bad suite/case/keyword/variable name warning
- show too little test cases in one suite warning
- show too many test cases in one suite warning
- show some dangerous keywords like (Fatal Error) usage warning (or error)
- show too many steps in one test case warning
- show too little steps in one test case warning
- show too many steps in one keyword warning
- show mandentory tags missing warning
- show too many arguments in one keyword warning
- show set suite variable/set test variable invalid usage warning
- show set global variable usage warning
- show deprecated keywords used warning
- show hard coded warning
- show case duplication warning
- show dry-run warning or errors
- show no __init__ file warning (neither setup nor teardown defined)
- use “Run Keyword and Ignore Error” but no return value used warning
- performance issue warning like using sleep
- complexity checking
- dependency checking
- recursive calling checking
In the same time, I found the operation I did was similar as a robot listener. And after reading the code of
RobotFramework, I found it has provided a class called
robot.result.visitor.SuiteVisitor, it helps to do the work to walk through the whole suite.
Based on this, the first checkpoint was ready, it is like below:
import os from robot.model import SuiteVisitor from robot import get_version if get_version() < '2.8.0': raise RuntimeError('RobotFramework 2.8+ required!') from robot.api import TestSuiteBuilder, TestSuite class NamingChecker(SuiteVisitor): def start_suite(self, _suite): if '.' in os.path.basename(_suite.source): print 'suite name should not contain "."' if __name__ == '__main__': paths = sys.argv[1:] if '-h' in paths or '--help' in paths: print __doc__ print 'usage:\n\npython rfexplain.py <path 1> <path 2> ...\n' sys.exit(0) suite = TestSuiteBuilder().build(*paths) suite.visit(NamingChecker())
In this script, it will check the name of suite, if it contains “.”, it will print a message. It is simple, but it can work.
At this moment, I wrote some test robot cases to verify the script, the developing was TDD, and agile. In that two days, the first workable version was finished, including most of the checkpoints. See the gist below:
If you have interest on this script, welcome to use and modify it. If you have good ideas about it, don’t forget to share with me :)