XSLTUnit with XSLT2.0 (and without exslt)

Recently whilst working on a little XSL project of mine, I came across a rather nifty unit testing suite for XSLT called XSLTUnit. It's a simple way of expressing test conditions for your XSL stylesheet by writing a test style sheet (which xsl:imports the stylesheet it's testing) and expressing a set of assertions. Now this is all well and good (in fact really quite neat) - but when I came to try to use it I came a cropper when using my favourite XSL sandpit Kernow, and also when using the xsl extensions to NetBeans. The error I got from Kernow (just so it's indexed on google) was:

net.sf.saxon.trans.XPathException: Cannot find a matching 1-argument function named {http://exslt.org/common}node-set(). There is no Saxon extension function with the local name node-set
Cannot find a matching 1-argument function named {http://exslt.org/common}node-set(). There is no Saxon extension function with the local name node-set

This is essentially because XSLTUnit uses the exslt extensions, whilst Kernow uses SaxonHE (which doesn't support exslt). Quite frankly I'm loathed to pay for SaxonPE/EE just for this, especilly as it restricts anyone else wanting to run the unit tests (which is a recipe for unit testing being ignored). Fortunately in the years since Eric van der Vlist wrote XSLTUnit in 2002, the wonders of XSLT2.0 have burst into the world, and hapilly XSLT2.0 treats results as nodes without the need for the node-set extension.

This meant it wasn't a hard job to ammend the XSLTUnit library and sample test sheet to XSLT2.0 and get it working in Kernow (and anything else which is XSLT2.0 aware (given that it came out in 2007 it isn't like it's bleeding edge). This tweaking basically involved taking the exsl attributes out of the stylesheet, changing the stylesheet version from 1.0 to 2.0, and changing "exsl:node-set($nodes1)" to "$nodes1" - it really was that easy.

Whilst I was at it, I made one other small change to the test xsl sheet (tst_library.xsl is the example from xsltunit.org). This was to apply the sheets to the standard xslt input stream rather than "document('library.xml')/<xpath>". I couldn't work out why you'd want to include the name of the input xml in the stylesheet itself as it just cuts down the flexibility to run your unit test with lots of input files. After that it all worked a treat.

I did send this back to Eric, but I don't know if he's actively maintaining xsltunit or if he wants to make it xslt 2. Therefore if it's of any use to anyone, I've put this all into a zip for anyone to download - everything except tst_library2.xsl and xslunit2.xsl are the originals from xsltunit.org. The two files with 2 at the end are merely alterations to the same files without the 2 suffix (as described above).

Licensing

As stated on the xsltunit site:

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ``Software''), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
    The name of the authors when specified in the source files shall be kept unmodified.

THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL ERIC VAN DER VLIST BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

My alternations are made in the same spirit, copyright of all the origional bits remain with Eric van der Vlist 2001, 2003.

Running SOAPui Tests from Jenkins/Hudson/Ant

The challenge

A while ago I came across a problem which I suspect has been tackled by a lot of people over the years: running one/many SOAPUI unit test packs from Jenkins. So far all the solutions I've seen from the web basically involve putting an exec call into your ant script, and pointing it at the testrunner cmd/sh script. This is fine and does work, but below is a slightly more complete solution which I hope is easy to just drop in and configure - plus I included some report generation stuff which is useful from a QA audit perspective if anyone wants to check what testing you've been doing.

Essentially this blog covers a simple ant build script which includes:

• A generic way of calling soapui from ant (using a re-usable ant macrodef)
• HTML report generation - generating a junit style html report on the result of your tests
• A command line so you can invoke the build script and enter the test to run (although the only reason you'd not do this from the soapUI GUI is if you wanted the html report for a single test suite).
• An “all” target (far more useful) which can be configured to run all the tests in the test suites, and can be invoked from Jenkins (which is sort of the point).

Getting Started

Before we get going I assume you have (or need to get) the following:

1. SoapUI: I originally wrote the script using soapUI 3.6.1 so I know it's good to go with that version and presumably any version since.
2. Apache Ant: If you wish to generate test reports you need version 1.5 or higher.
3. AntContrib: v0.3 or higher to be added to your ant lib folder.

To start with there's a simple zip of stuff to be working with. It includes:

1. build.xml - A sample ant build file – this is the crux of everything we'll look at.
2. VerySimpleProject – a soap UI project with 2 test suites
3. VerySimpleProject2 – another soap UI project with a single test suite
4. junit-noframes.xsl – a modified verison of the xsl which comes with ant but has been altered to make it work better with the SOAPUI

Making build.xml your own

Configuuring Properties

The xml file has a number of properties which need to be configured specific to your environment and/or project. They are:

• jar.soapui: The name of the soapui JAR file, which is sadly version specific
• dir.soapui: Self explanatory really, this is the directory soapui is in on the machine running ant
• dir.temp: a directory to store temporary files created during the execution. This is created upon build and deleted once the run has completed, so make sure it's specific to this purpose and not c:\. Generally ./temp is good enough so shouldn't need to be changed
• dir.reports: a directory where the report outputs are stored (this needs to exist before the script is run)
• dir.xsl: the directory the custom xsl for generating reports is stored in, this is by default ./xsl which shouldn't need to be altered
• java.executable: the full path to java on this machine
• test.report.title: The title placed at the top of your output report (passed as a parameter to the report generating xsl)

Configuring build all

Once these are set, you're nearly ready to go. All you need to do is say which tests to run. The best way of doing this is to put all the tests in the "all" target and then ant will invoke them all each time. Due to the naming convention I was using at the time this assumes a default test suite called "Full Test" and as such if you don't specify a suite name it will default to that. If you want to call other test suites you just need to name them. This can be seen in the all target example below:

So this calls:

1. clean - a general tidy up target and creation of the temporary directory (dir.temp)
2. runTest macro - as no suiteName property is passed to defaults to "Full Test".
3. calls runTest with a different suite from the same sample project
4. calls runTest with a suite from a second soapui project
5. calls the report target. The report target:
• generates a single html report covering all the tests which have been run (since the last clean - step 1 in this case) 
• zips up the test runner output log/error files - in case a test fails and you want to look at these detailed logs
• Sets the names of both the zip and the html to include the date and time the ant script started running - this is just so subsequent runs don't overwrite it
6. calls tidyup to delete the log directory and a couple of the soapui files left over from the job.

Default target

The default target of the build.xml is "test" which will expect keyboard inputs for the name of the project file (without the .xml on the end) and the test suite. This can occasionally be useful if you want to test and report on a single report, however you might find it more useful to set the project default from test to all.

Really that's all you need to know to get this up and running... but just for completeness I'll do a quick tour of the runTest macro and the reports target if you're interested in understanding the nuts and bolts. If not then you're ready to go.

What's under the covers?

Run Test

The run test macro essentially invokes soapui, but rather than calling the cmd/sh script to do it it does it directly (hence the need to know where java was in the properties steps above). Most of this is quite obvious from looking at the code and you can play with the output, and the max memory should you need to. 

The macro does pass a number of arguments to soapUI, which are explained on their website. Of these we only use:

• a to ensure verboseness in the output of errors
• j to tell soapui to generate the xml which gets turned into the html report
• s to tell it the name of the test suite to run
• f to say where to put the outputs

Optionally, if you have any properties which are used in your tests (in our case we had a jdbc connection string as a property) you can get ant to pass these in, that way you can externalise these for running on different environments without the need to change your test scripts. These can be set for the developers environment within the soapUI project, and then ant will merely overwrite them on the continuous integration environment - cool huh?

Finally there's some tidy up in renaming the outputs to ensure that two project files with the same test suite name (e.g. Full Test) won't overwrite each other.

Reports 

The reports target is very simple, which is largely because it uses the standard junitreport functionality, and because most of the work I did here was hacking the xslt to be more soapUI friendly (describing that is too big a job for today). 

Essentially it grabs the xml files for all the test suites run since last clean (each renamed to be unique by the runTest macro above) and runs the junit-noframes.xsl on them. Unlike the standard ant one, the xsl now has three parameters:

• runDateTime (because unlike java SoapUI doesn't include this in the output xml)
• resultsFile - which means the html can have some text to say which zip contains the full logs
• title - the title of the html report

Finally, all the logs are zipped into a file in the reports folder, and the html is moved from the temp folder to the reports folder, as it would be a shame for the tidyup job to delete it after we've only just created it.

Licensing

...and finally: 

The junit-noframes.xsl is under the apache 2.0 license - and my updates to it fall under the same.

The custom build.xml is open source under the GPLv3 license - hope it's useful.