Jenkins: Testing with post conditions in your pipeline

The behaviour of your pipeline can be much more complex than the simple success/failure flow shown in the previous blog post.
The Jenkins declarative pipeline syntax has a lot more ways to decide whether to execute a step or not, and this also requires a proper test scenario.

This post is part of a Series in which I elaborate on my best practices for running Jenkins at scale which might benefit Agile teams and CI/CD efforts.

This post will focus on the post section, which is very much like a regular stage but is only executed when specific requirements are met.

Our pipeline

We’ll be working with the pipeline from our previous posts, as it included a post section on the first stage. It’s not the first time it is shown here, so you might be familiar with it.

pipeline {
    agent none

    stages {
        stage('Build and Unit test') {
            agent { label 'maven' }
            steps {
                script {
                    module_Maven('clean verify')
                }
            }
            post {
                always {
                    junit testResults: '**/target/surefire-reports/*.xml', allowEmptyResults: false
                }
            }
        }
        stage('Publish to Nexus') {
            agent { label 'maven' }
            steps {
                script {
                    echo 'This is where we publish to Nexus'
                    module_Artifact.publish()
                }
            }
        }
    }
    post {
        always {
            script {
                module_Notification.sendEmail(currentBuild.result)
            }
        }
    }
}

The important part here is that there actually are two post sections in this pipeline, on after the first stage and the other on a global level.
They both are evaluated by Jenkins at separate moments, either after the execution of that specific stage, or after execution (or skipping) of all stages.

From the perspective of writing your tests for either of these scenarios, this doesn’t matter. We have to use the same method for both, it is just the moment that differs.
Spoiler: it is actually very easy with a little help from Spock!

Post conditions

There are a lot of post conditions, here’s a small list:

Our testing frameworks support them all, so we should be able to validate that in our test.

Testing the conditions

I’ll focus on the post conditions of the first stage for now. This is the relevant snippet of our pipeline:

stage('Build and Unit test') {
    agent { label 'maven' }
    steps {
        script {
            module_Maven('clean verify')
        }
    }
    post {
        always {
            junit testResults: '**/target/surefire-reports/*.xml', allowEmptyResults: false
        }
    }
}

A very barebones test for this snippet (without anything related to the post conditions) would be something like this:

import groovy.testSupport.PipelineSpockTestBase
import module_Artifact
import module_Notification

class minimalTest extends PipelineSpockTestBase {

    def script
    def mavenMock
    def artifactMock
    def notificationMock

    def setup() {
        registerMocks()
        registerPluginMethods()
        script = loadScript('vars/defaultPipeline.groovy')
    }

    def cleanup() {
        printCallStack()
    }

    void 'Happy flow'() {
        when:
        script.call([:])

        then:
        1 * mavenMock.call('clean verify')
        assertJobStatusSuccess()
    }

    def registerPluginMethods() {
        // Junit
        // https://plugins.jenkins.io/junit
        helper.registerAllowedMethod('junit', [HashMap.class], null)
    }

    def registerMocks() {
        mavenMock = Mock(Closure)
        helper.registerAllowedMethod('module_Maven', [String.class], mavenMock)

        artifactMock = Mock(module_Artifact)
        binding.setVariable('module_Artifact', artifactMock)

        notificationMock = Mock(module_Notification)
        binding.setVariable('module_Notification', notificationMock)
    }
}

As you can see, we check for the occurrence of a call on our Maven mock, and conclude everything went fine after that. There is no check for the call on junit (which resides inside our post condition). Yet!

To add this assertion, we need to add a Mock to the mix. The revised code is below:

void 'Happy flow'() {
    given:
    def junitMock = Mock(Closure)
    helper.registerAllowedMethod('junit', [HashMap.class], junitMock)

    when:
    script.call([:])

    then:
    1 * mavenMock.call('clean verify')
    1 * junitMock.call(_)
    assertJobStatusSuccess()
}

This will register a new allowed method, which precedes over the one registered in the method registerMocks().

When we run this test, the callstack would be very similar to this:

>>>>>> pipeline call stack -------------------------------------------------
   defaultPipeline.call({})
      defaultPipeline.pipeline(groovy.lang.Closure)
         defaultPipeline.agent(groovy.lang.Closure)
         defaultPipeline.stages(groovy.lang.Closure)
            defaultPipeline.stage(Build and Unit test, groovy.lang.Closure)
               defaultPipeline.agent(groovy.lang.Closure)
                  defaultPipeline.label(maven)
               defaultPipeline.steps(groovy.lang.Closure)
                  defaultPipeline.script(groovy.lang.Closure)
                     defaultPipeline.module_Maven(clean verify)
               defaultPipeline.post(groovy.lang.Closure)
                  defaultPipeline.always(groovy.lang.Closure)
                     defaultPipeline.junit({testResults=**/target/surefire-reports/*.xml, allowEmptyResults=false})
...

This shows us the post section is executed, and the always condition is evaluated after which the junit step is run and completed.
Also, the test found that there was 1 execution on the junit Mock, so all is well here!

In a rainy day scenario in which the call to maven yields an error, we expect the always steps to be run anyway. It should be always, remember?
To verify this, one could use test code like this:

void 'A maven failure should still interpret the junit test report'() {
    given:
    def junitMock = Mock(Closure)
    helper.registerAllowedMethod('junit', [HashMap.class], junitMock)

    when:
    script.call([:])

    then:
    1 * mavenMock.call('clean verify') >> { binding.getVariable('currentBuild').result = 'FAILURE' }
    1 * junitMock.call(_)
    assertJobStatusFailure()
}

Stage 'Publish to Nexus' skipped - job status: 'FAILURE'
>>>>>> pipeline call stack -------------------------------------------------
   defaultPipeline.call({})
      defaultPipeline.pipeline(groovy.lang.Closure)
         defaultPipeline.agent(groovy.lang.Closure)
         defaultPipeline.stages(groovy.lang.Closure)
            defaultPipeline.stage(Build and Unit test, groovy.lang.Closure)
               defaultPipeline.agent(groovy.lang.Closure)
                  defaultPipeline.label(maven)
               defaultPipeline.steps(groovy.lang.Closure)
                  defaultPipeline.script(groovy.lang.Closure)
                     defaultPipeline.module_Maven(clean verify)
               defaultPipeline.post(groovy.lang.Closure)
                  defaultPipeline.always(groovy.lang.Closure)
                     defaultPipeline.junit({testResults=**/target/surefire-reports/*.xml, allowEmptyResults=false})
            defaultPipeline.stage(Publish to Nexus, groovy.lang.Closure)
         defaultPipeline.post(groovy.lang.Closure)
            defaultPipeline.always(groovy.lang.Closure)
               defaultPipeline.script(groovy.lang.Closure)

This shows us that the junit step is still executed, even after the failure of the maven call. So, we’re in the clear now!

Testing a condition which requires a previous run

Testing conditions like always, cleanup or any of the end states like successful, failure or unstable is quite easy when you’re using the method shown previously.
The tests for other conditions that require a previous run are a little different, but luckily not much more complicated. The trick here is to not only set the execution status of the current run, but also the status of the one before that.

Let’s modify our pipeline to include a changed post condition like in the snippet below:

stage('Build and Unit test') {
    agent { label 'maven' }
    steps {
        script {
            module_Maven('clean verify')
        }
    }
    post {
        always {
            junit testResults: '**/target/surefire-reports/*.xml', allowEmptyResults: false
        }
        changed {
            module_Notification.sendEmail(currentBuild.result)
        }
    }
}

The pipeline now sends an extra notification when the status of the call to Maven changes (e.g. from a failure to a successful run).

We’ve seen previously that you need to set the variable currentBuild.result to influence the execution status of the current run. To set the status of the previous run, you just need to set currentBuild.previousBuild.status! In code:

binding.getVariable('currentBuild').result = <RESULT>
binding.getVariable('currentBuild').previousBuild.result = <PREVIOUS_RESULT>

This can be done at any moment in time during the execution of your tests, so they can be set in the setup, before each test, during a specific test or any other moment you see fit.

A test for the pipeline as it stands now could look something like this:

void 'Send notification when status of maven call changes'() {
    given:
    def junitMock = Mock(Closure)
    helper.registerAllowedMethod('junit', [HashMap.class], junitMock)

    and:
    binding.getVariable('currentBuild').previousBuild.result = 'FAILED'

    when:
    script.call([:])

    then:
    1 * mavenMock.call('clean verify')
    1 * junitMock.call(_)
    2 * notificationMock.sendEmail(_)
    assertJobStatusSuccess()
}

A previous run is by default considered to have been successfully completed, so we will have to set it to a specific value for it to have been “changed”. Also: the current run is defaulted to “successful”, but this is usually what we want anyway.

The callstack for this testrun would be:

>>>>>> pipeline call stack -------------------------------------------------
   defaultPipeline.call({})
      defaultPipeline.pipeline(groovy.lang.Closure)
         defaultPipeline.agent(groovy.lang.Closure)
         defaultPipeline.stages(groovy.lang.Closure)
            defaultPipeline.stage(Build and Unit test, groovy.lang.Closure)
               defaultPipeline.agent(groovy.lang.Closure)
                  defaultPipeline.label(maven)
               defaultPipeline.steps(groovy.lang.Closure)
                  defaultPipeline.script(groovy.lang.Closure)
                     defaultPipeline.module_Maven(clean verify)
               defaultPipeline.post(groovy.lang.Closure)
                  defaultPipeline.always(groovy.lang.Closure)
                     defaultPipeline.junit({testResults=**/target/surefire-reports/*.xml, allowEmptyResults=false})
                  defaultPipeline.changed(groovy.lang.Closure)
            defaultPipeline.stage(Publish to Nexus, groovy.lang.Closure)
               defaultPipeline.agent(groovy.lang.Closure)
                  defaultPipeline.label(maven)
               defaultPipeline.steps(groovy.lang.Closure)
                  defaultPipeline.script(groovy.lang.Closure)
                     defaultPipeline.echo(This is where we publish to Nexus)
         defaultPipeline.post(groovy.lang.Closure)
            defaultPipeline.always(groovy.lang.Closure)
               defaultPipeline.script(groovy.lang.Closure)

I’ll leave the rest of the post conditions for you to figure out yourself, but the way of working basically stays the same: when they are in your pipeline you can (and should!) write assertions for them based on the outcome of the steps before.