The recent posts on this blog about Jenkins have been preparing us for this: validation of the behaviour of your Jenkins Pipeline in a test, in order to be able to check what the impact is of any changes you make to this pipeline, before it lands on your live instance and possibly influences the teams that are working with it.
This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.
So far, we’ve seen how to set up your project in your IDE, how to run a pipeline from your Shared Library, and how to write your first (simple) test for this pipeline.
This post elaborates on how to expand this test to validate steps and to check for the behaviour of your pipeline when something fails.
Recap
Remember our simple pipeline and test? If you don’t, here are two excerpts to show the relevant parts we will focus on:
stage('Build and Unit test') {
agent { label 'maven' }
steps {
script {
module_Maven('clean verify')
}
}
...
void 'Happy flow'() {
when:
script.call([:])
then:
assertJobStatusSuccess()
}
Checking for calls on steps
One of the first steps in the pipeline, is to do a call to a module called module_Maven, which in turn actually does a callout to maven to perform some actions on the checked out code.
We want to check for the occurrence of this call, and validate that the call does a clean verify.
Remember we already have created a Mock-object for this?
That will come in very handy, as the Spock framework enables us to count the number of calls on a Mock and even lets us check for values in the call!
Spock calls this ‘interaction based testing’ and has a very particular syntax.
then:
1 * mavenMock.call('clean verify')
This code would translate to natural language as: “Check for a call on our maven mock with parameters ‘clean verify’. It should happen only once!”
Let’s take a closer look at the then: block. It contains two interactions, each of which has four distinct parts: a cardinality, a target constraint, a method constraint, and an argument constraint:
1 * mavenMock.call('clean verify')
| | | |
| | | argument constraint
| | method constraint
| target constraint
cardinality
This single line of code checks for one (1) call to maven where the parameter equals ‘clean verify’.
When this single call does not occur, Spock will warn you about this and the test will fail.
If we were to modify the maven call in the pipeline to module_Maven('clean install') and run the test, the output would be very similar to this:
Too few invocations for:
1 * mavenMock.call('clean verify') (0 invocations)
Unmatched invocations (ordered by similarity):
1 * mavenMock.call([<java.lang.String@efe71ec4 value=clean install hash=-270065980>])
One or more arguments(s) didn't match:
0: argument == expected
| | |
| | clean verify
| false
[<java.lang.String@efe71ec4 value=clean install hash=-270065980>]
...
As you can see, Spock complains that the arguments it expected on the call, are not equal to what it received, and that it therefore didn’t count enough invocations which match the constraints. The test failed because of this.
Spock has a very detailed manual on Interaction Based Testing, which also lists all other possibilities in wildcarding and constraints. Please read this if you want to know more about that!
Happy flow test
The pipeline we’re testing here has multiple invocations which occur when the pipeline should run successfully. We’d like to check them all, and when we do, it would look similar to this:
void 'Happy flow'() {
when:
script.call([:])
then:
1 * mavenMock.call('clean verify')
1 * artifactMock.publish()
1 * notificationMock.sendEmail(_)
assertJobStatusSuccess()
}
The result would be:
minimalTest > Happy flow PASSED
Yay!
I snuck in a little extra feature of Spock, I used a wildcard for the argument constraint for the call on the notificationMock. Spock’s version of a wildcard is an underscore (_) as the asterisk (*) is already in use for the notation of the cardinality.
Rainy day test
Usually, everything is running fine, but our pipeline also has some logic in it which deals with failure of a step. For instance: when the maven build fails it will gather the junit test results, but will not run the next stage(s).
Naturally, we’d love to check this in a test.
So how do we make the Maven build fail during a specific test? This is where the awesome power of Spock (with a little help of Byte-Buddy and Objenesis) mocking and stubbing comes in.
Spock enables us to define the behaviour of a Stub, and we will use this to throw an error at a specified moment during the test.
Stubbing, by default, works like this:
mavenMock.call(_) >> "ok" | | | | | | | response generator | | argument constraint | method constraint target constraint
In natural language, it would read as: “Whenever maven is called, return the value ‘ok’”.
Stubbing and Mocking can conveniently be combined to provide a very powerful way of defining the behaviour of your Mock/Stub and validate your pipeline all at the same time.
It would look something like this:
1 * mavenMock.call(_) >> "ok" | | | | | | | | | response generator | | | argument constraint | | method constraint | target constraint cardinality
And in our test, where a call to Maven would yield an error, we do not just return a value from our Stub, but we have to set a value in our test framework as a side effect with a little help from a Closure:
then:
1 * mavenMock.call(_) >> {
binding.getVariable('currentBuild').result = 'FAILURE'
}
This is the Spock way of making the framework expect 1 call on the Maven mock/stub where the parameters don’t really matter (as long as just one parameter is passed). If this happens, the Closure is executed which in turn sets the Variable currentBuild to the value ‘FAILURE’.
Finally, this is picked up by the pipeline which interprets it as a failure and marks that step as FAILED and it proceeds accordingly.
When this happens, we want to validate that the artifact does not get published to our Binary Repository Manager, and that the user is notified of the result of this pipeline.
All of this combined would look something like this:
void 'Rainy day'() {
when:
script.call([:])
then:
1 * mavenMock.call(_) >> {
binding.getVariable('currentBuild').result = 'FAILURE'
}
0 * artifactMock.publish()
1 * notificationMock.sendEmail(_)
assertJobStatusFailure()
}
And a run would yield:
minimalTest > Happy flow PASSED minimalTest > Rainy day PASSED
On demand Mocking and Stubbing
In our previous example, the Rainy Day scenario only checked for interactions on ‘our own’ pipeline steps, but did not check on pipeline steps which come preinstalled with Jenkins.
To enable us to do so, we have to create a Mock or Stub during the test, on demand.
To do so, we specify our Mock in the given clause of our Spock test:
void 'Rainy day'() {
given:
def junitMock = Mock(Closure)
helper.registerAllowedMethod('junit', [HashMap.class], junitMock)
when:
script.call([:])
...
We register a Mock like any other step with an implicit call and specify a HashMap as it’s parameter.
The given is executed during the setup-phase of this specific test before the when clause is run. As this is local to our test, this Mock will only be available to this test and not to any before or after. This enables us to define behaviour very specific to a single test, where our Stubs will not interfere with other tests.
Our revised Rainy Day test would look something like this:
void 'Rainy day'() {
given:
def junitMock = Mock(Closure)
helper.registerAllowedMethod('junit', [HashMap.class], junitMock)
when:
script.call([:])
then:
1 * mavenMock.call(_) >> {
binding.getVariable('currentBuild').result = 'FAILURE'
}
1 * junitMock.call(_)
0 * artifactMock.publish()
1 * notificationMock.sendEmail(_)
assertJobStatusFailure()
}
The test above checks for the occurrence of a call to interpret the junit test results genrated by our maven call, and will fail if it is not run. As the junit call is specified inside an always block which is inside the post block of the stage, it should always be executed, regardless of the outcome of the steps inside the stage.
Next steps
I’ve shown you several common examples of how to use Spock and the other test frameworks for declarative Jenkins pipelines and this should enable you to start writing your own tests for your pipelines!
In the next post, I’ll try to find some more obscure examples and perform some Groovy/Spock trickery which might come in handy at a later moment in time!
