Cache Unit Test with jenkins

Hi Javier,

There are a few topics for running builds and unit tests via Jenkins (or really any CI tool):

• Calling in to Caché (or IRIS; the approaches are very similar)
• Reporting unit test results
• Test coverage measurement and reporting

Here's a quick intro; if you have questions on any details I can drill down further.

Calling in to Caché:

The most common approach I've seen is writing out to a file and then using that as input to csession / iris session. You can see some examples of this (for IRIS, with containers, but quite transferrable) here: https://github.com/timleavitt/ObjectScript-Math/blob/master/.travis.yml - I'm planning to write an article on this soon.

Some rules for this:

• Either enable OS authentication or put the username/password for the build user in the script or environment variables
• End the script with Halt (in case of success) or $System.Process.Terminate($Job,1) (to signal an OS-level error you can pick up from errorlevel/etc.); alternatively, always end with Halt and create a "flag file" in the case of error, the existence of which indicates that the build failed.
• Keep the script short - ideally, put the meat of the build logic in a class/routine that is loaded at the beginning, then run that.

Sample for Windows:

:: PREPARE OUTPUT FILE
set OUTFILE=%SRCDIR%\outFile
del "%OUTFILE%"

:: NOW, PREPARE TO CALL CACHE
::
:: Login with username and password
ECHO %CACHEUSERNAME%>inFile
echo %CACHEPASSWORD%>>inFile

:: MAKE SURE LATEST JENKINS BUILD CLASS HAS BEEN LOADED
echo do $system.OBJ.Load("","cb") >>inFile

:: RUN JENKINS BUILD METHOD
echo do ##class(Build.Class).JenkinsBuildAndTest("%WORKSPACE%") >>inFile

:: THAT'S IT
echo halt >>inFile

:: CALL CACHE
csession %INSTANCENAME% -U %NAMESPACE% <inFile

echo Build completed. Press enter to exit. :: PAUSE

pause > nul

:: TEST IF THERE WAS AN ERROR
IF EXIST "%OUTFILE%" EXIT 1

:: Clear the "errorlevel" variable that (it looks like) csession sets, causing successful builds to be marked as failure
(call )

Sample for Linux:

# PREPARE OUTPUT FILE
OUTFILE=${WORKSPACE}/outFile
rm -f $OUTFILE

# PREPARE TO CALL IRIS
# Login with username and password
echo $IRISUSERNAME > infile.txt
echo $IRISPASSWORD >> infile.txt

# MAKE SURE LATEST JENKINS BUILD CLASS HAS BEEN LOADED
echo 'do $system.OBJ.Load("'${WORKSPACE}'/path/to/build/class"),"cb")' >>infile.txt

# RUN JENKINS BUILD METHOD
echo 'do ##class(Build.Class).JenkinsBuildAndTest("'${WORKSPACE}'")' >>infile.txt

# THAT'S IT
echo halt >> infile.txt

# CALL IRIS
# csession is the equivalent for Caché
iris session $IRISINSTANCE -U $NAMESPACE < infile.txt

# TEST IF THERE WAS AN ERROR
if [ -f $OUTFILE ] ; then exit 1 ; fi

The next question is, what does Build.Class do? Given the Jenkins workspace root (WORKSPACE variable), it should load the code appropriately (likely after blowing away the code database to start with a clean slate; %Installer can help with this), then set ^UnitTestRoot based on the workspace directory, then run the tests, then report on results. Best to wrap the whole thing in a Try/Catch and throw/handle exceptions appropriately to ensure the error flag file / exit code is set.

Reporting Unit Test Results:

See https://github.com/intersystems-community/zpm/blob/master/src/cls/_ZPM/PackageManager/Developer/UnitTest/JUnitOutput.cls
(feel free to copy/rename this if you don't want the whole community package manager) for a sample of a jUnit export; Jenkins will pick this up and report on it quite easily. Just pass an output filename to the method, then add a post-build action in Jenkins to pick up the report. (You'll want to call this from your build script class.)

Measuring Test Coverage:

Seeing how much of your code is covered by unit tests helps to close the feedback loop and enable developers to write better tests - I presented on this at Global Summit a few years ago. See https://openexchange.intersystems.com/package/Test-Coverage-Tool - we've successfully used this with Jenkins for both HealthShare and internal applications at InterSystems. It can produce reports in the Cobertura format, which Jenkins will accept. Instead of using %UnitTest.Manager, call TestCoverage.Manager. The parameters detailed in the readme can be passed into the third argument of RunTest as subscripts of an array; to produce a Cobertura-style export (including reexporting all source in UDL for coverage reporting in the Jenkins UI), add a "CoverageReportFile" subscript pointing to an appropriate place in the Jenkins workspace, and set the "CoverageReportClass" subscript to "TestCoverage.Report.Cobertura.ReportGenerator".

If you want to use the Jenkins coverage/complexity scatter plot, use https://github.com/timleavitt/covcomplplot-plugin rather than the original; I've fixed some issues there and made it a bit more resilient to some oddities of our Cobertura-style export (relative to the data Cobertura actually produces).