Parallel, Matrix, input and Deploy Stages in Jenkins

Version note: The examples in this post were run with Jenkins 2.580.2 LTS (Long-Term Support), the Lockable Resources plugin 1582, Pipeline: Milestone Step, the SSH (Secure Shell) Agent plugin, Debian 13 containers running systemd, and .NET 10. The IIS (Internet Information Services) deploy was not run on this machine.

Guess First

Two builds of a pipeline that deploys to staging start almost at the same time: first #4 for commit a1b2c3d, two seconds later #5 for the newer commit e4f5a6b. The deploy step is protected with lock('bookstore-staging'), so only one of them can deploy at a time. When both are done, which commit is running on staging? The lock guarantees deploying "one after the other", but in which order?

Bookstore's Deploy Flow

In the previous post the pipeline produced a single bookstore-1.4.N.tgz package. Now we take that package first to staging and, after a human's approval, to production. The part of the Jenkinsfile after Package:

stage('Deploy to staging') {
    when { branch 'main' }
    agent { label 'docker' }
    steps {
        lock('bookstore-staging') {
            unstash 'package'
            sshagent(credentials: ['deploy-ssh-key']) {
                sh 'ci/deploy.sh jenkwork-staging "$VERSION" "bookstore-$VERSION.tgz"'
            }
        }
    }
}
stage('Approve production') {
    when { branch 'main' }
    options {
        timeout(time: 1, unit: 'HOURS')
    }
    input {
        id 'ApproveProduction'
        message 'Deploy this version to production?'
        ok 'Deploy'
        submitter 'release-manager'
    }
    steps {
        milestone(ordinal: 10, label: 'production approved')
    }
}
stage('Deploy to production') {
    when { branch 'main' }
    agent { label 'docker' }
    steps {
        lock('bookstore-production') {
            unstash 'package'
            sshagent(credentials: ['deploy-ssh-key']) {
                sh 'ci/deploy.sh jenkwork-prod "$VERSION" "bookstore-$VERSION.tgz"'
            }
        }
    }
}

Besides archiving the package, the Package stage also keeps it with stash name: 'package'; the deploy stages unstash the same file. So the package that was tested and the package that goes to staging and production are the same bytes; nothing is rebuilt during the deploy. when { branch 'main' } limits all three stages to main: PR (Pull Request) builds never touch the deploy credentials.

input and agent none: Not Holding an Executor While Waiting for Approval

The "Approve production" stage has no agent. This detail matters: imagine a build waiting hours for approval, maybe until the end of the week. If the stage were on an agent, an executor would stay locked for nothing during that wait. Because the pipeline has agent none at the top level and this stage takes no agent of its own, the wait is only a lightweight record on the controller. The input directive in Declarative is also evaluated before the stage's agent; even in a stage with an agent, no executor is taken before the approval.

submitter 'release-manager' limits who (which user or group) can approve; options { timeout(...) } aborts the build if no approval comes within an hour, so waiting builds do not pile up forever. When the build reaches the approval point, the log says:

[2026-11-08T04:42:45.765Z] Lock released on resource [Resource: bookstore-staging], resources: [bookstore-staging]
[2026-11-08T04:42:45.816Z] Timeout set to expire in 1 hr 0 min
[2026-11-08T04:42:45.830Z] Deploy this version to production?
[2026-11-08T04:42:45.830Z] Deploy or Abort

You approve with the button in the UI (User Interface); I called the input's REST (Representational State Transfer) endpoint as the release-manager user: POST /job/bookstore/job/main/5/input/ApproveProduction/proceedEmpty. Giving an id makes this URL (Uniform Resource Locator) predictable; if you do not, Jenkins generates an id from the message. After the approval:

[2026-11-08T04:43:03.115Z] Approved by release-manager
[2026-11-08T04:43:03.132Z] Trying to pass milestone 10
[2026-11-08T04:43:03.170Z] Running on docker-agent-1 in /home/jenkins/agent/workspace/bookstore_main
[2026-11-08T04:43:03.231Z] Trying to acquire lock on [Resource: bookstore-production]
[2026-11-08T04:43:03.232Z] Lock acquired on [Resource: bookstore-production], resources: [bookstore-production]
[2026-11-08T04:43:03.522Z] + ci/deploy.sh jenkwork-prod 1.4.5 bookstore-1.4.5.tgz
[2026-11-08T04:43:03.522Z] Warning: Permanently added 'jenkwork-prod' (ED25519) to the list of known hosts.
[2026-11-08T04:43:03.522Z] curl: (7) Failed to connect to localhost port 5000 after 0 ms: Could not connect to server
[2026-11-08T04:43:04.873Z] {"status":"Healthy"}
[2026-11-08T04:43:04.889Z] Lock released on resource [Resource: bookstore-production], resources: [bookstore-production]
Finished: SUCCESS

The "Approved by release-manager" line stays in the build's permanent record: the answer to who approved and when is here. After the approval the stage takes an agent again and checks out the same commit into the workspace.

Deploying to Linux: sshagent, rsync and systemctl

The deploy logic is not in the Jenkinsfile but in a small script in the repository. That improves readability and also makes it possible to try the script without Jenkins:

#!/usr/bin/env bash
# usage: ci/deploy.sh <host> <version> <package.tgz>
set -euo pipefail
HOST="$1"; VERSION="$2"; PACKAGE="$3"
SSH_OPTS="-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=$WORKSPACE/.known_hosts"
RELEASE="/opt/bookstore/releases/$VERSION"

rsync -az -e "ssh $SSH_OPTS" "$PACKAGE" "deploy@$HOST:/tmp/$PACKAGE"
ssh $SSH_OPTS "deploy@$HOST" bash -s -- "$RELEASE" "/tmp/$PACKAGE" <<'REMOTE'
set -euo pipefail
mkdir -p "$1"
tar -xzf "$2" -C "$1"
ln -sfn "$1" /opt/bookstore/current
rm -f "$2"
sudo systemctl restart bookstore-api
for i in $(seq 1 20); do
    if curl -fsS http://localhost:5000/health; then echo; exit 0; fi
    sleep 1
done
echo "health check failed" >&2
exit 1
REMOTE

Each version is unpacked under /opt/bookstore/releases/<version>, and the current symlink is switched to the new folder. Going back is as easy as pointing the symlink at the previous version and restarting the service. On the server side the deploy user's sudo rights are limited to two commands: systemctl restart bookstore-api and systemctl is-active bookstore-api. The application itself runs as a separate bookstore user with this systemd unit:

[Unit]
Description=Bookstore API
After=network.target

[Service]
User=bookstore
WorkingDirectory=/opt/bookstore/current
ExecStart=/usr/share/dotnet/dotnet /opt/bookstore/current/Bookstore.Api.dll
Environment=ASPNETCORE_HTTP_PORTS=5000
Environment=DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

The staging and production servers in this post are two Debian 13 containers running systemd; on a real VM (Virtual Machine) the flow is the same. The curl: (7) Failed to connect line in the log is the first attempt of the health check loop: the port is closed for a moment while the service restarts, and a second later {"status":"Healthy"} arrives. Without a health check, "deploy succeeded" would only mean "the files were copied". The staging server itself after the deploy:

$ ls -l /opt/bookstore/
lrwxrwxrwx 1 deploy deploy   29 Nov  8 04:42 current -> /opt/bookstore/releases/1.4.5
drwxr-xr-x 1 deploy deploy 4096 Nov  8 04:42 releases
$ curl -s localhost:5000/api/books/1
{"id":1,"title":"Domain-Driven Design","author":"Eric Evans","price":54.90,"stock":12}
$ curl -s localhost:5000/ | grep -o "<title>[^<]*</title>"
<title>BookstoreWeb</title>
$ systemctl is-active bookstore-api
active

The API (Application Programming Interface) and the Angular app are served from the same package and the same process. We will discuss the details of the systemd unit (restart policies, journald, security settings) in the upcoming Web Servers and Hosting series post "Running a .NET App as a systemd Service on Linux".

Lockable Resources and milestone

The answer to the opening question. First, I triggered a small job that uses only lock twice in a row with two different commit parameters.

Build #4 (commit a1b2c3d):

Trying to acquire lock on [Resource: bookstore-staging]
The resource [bookstore-staging] is locked by build lock-demo #5 #5 since Nov 8, 2026, 4:47 AM.
[Resource: bookstore-staging] is not free, waiting for execution ...
Lock acquired on [Resource: bookstore-staging], resources: [bookstore-staging]
Build 4 deploys a1b2c3d to staging

Build #5 (commit e4f5a6b):

Lock acquired on [Resource: bookstore-staging], resources: [bookstore-staging]
Build 5 deploys e4f5a6b to staging

#4 was triggered earlier, but #5 got the lock first; #4 waited and deployed after #5. The result: the old commit a1b2c3d is running on staging. The lock solves concurrency, not ordering. The tool for ordering is milestone: when a build passes a milestone, older builds that have not yet passed it are cancelled. Used together with inversePrecedence: true on the lock (the newest waiting build gets the lock first):

lock(resource: 'bookstore-staging', inversePrecedence: true) {
    milestone(ordinal: 1, label: 'deploy staging')
    echo "Build ${BUILD_NUMBER} deploys ${params.COMMIT} to staging"
    sleep time: 10, unit: 'SECONDS'
}

Build #2 (commit 1111111):

Trying to acquire lock on [Resource: bookstore-staging]
The resource [bookstore-staging] is locked by build lock-milestone #3 #3 since Nov 8, 2026, 4:48 AM.
[Resource: bookstore-staging] is not free, waiting for execution ...
Superseded by lock-milestone#3
Finished: NOT_BUILT

Build #3 (commit 2222222):

Lock acquired on [Resource: bookstore-staging], resources: [bookstore-staging]
Trying to pass milestone 1
Build 3 deploys 2222222 to staging

Build #4 (commit 3333333):

[Resource: bookstore-staging] is not free, waiting for execution ...
Lock acquired on [Resource: bookstore-staging], resources: [bookstore-staging]
Trying to pass milestone 1
Build 4 deploys 3333333 to staging

The oldest build (#2, commit 1111111) was cancelled as NOT_BUILT, and the newest build deployed last. In Bookstore's Jenkinsfile, milestone(ordinal: 10) right after the production approval does this: once a newer version is approved and passes, older builds still waiting for approval cannot go to production. One detail: if the resource named in lock is not defined, the plugin creates it temporarily (the "did not exist. Created." in the log); defining permanent resources and their labels with JCasC (Jenkins Configuration as Code) is tidier.

parallel, failFast and matrix

Running the checks before the deploy in parallel shortens the duration. failFast true stops the other branches as soon as one breaks:

stage('Checks') {
    failFast true
    parallel {
        stage('API tests') {
            agent { label 'docker' }
            steps {
                sh 'echo "a test failed"; exit 1'
            }
        }
        stage('Web tests') {
            agent { label 'docker' }
            steps {
                sh 'sleep 60'
            }
        }
    }
}

+ echo a test failed
a test failed
+ exit 1
+ sleep 60
Failed in branch API tests
Sending interrupt signal to process
Terminated
Failed in branch Web tests
ERROR: script returned exit code 1
Finished: FAILURE

The 60-second branch was terminated as soon as the first branch broke. options { parallelsAlwaysFailFast() } gives the same behavior for the whole pipeline.

matrix repeats the same stages for the values of an axis. Since Angular 22 supports both Node 22 and 24, running the web tests on both makes sense:

stage('Web on Node versions') {
    matrix {
        axes {
            axis {
                name 'NODE_VERSION'
                values '22', '24'
            }
        }
        agent {
            docker {
                image "node:${NODE_VERSION}-trixie"
                label 'docker'
                args '-v bookstore-npm:/npm'
            }
        }
        environment {
            npm_config_cache = '/npm'
        }
        stages {
            stage('Test') {
                steps {
                    git url: 'http://jenkwork-gitea:3000/bookstore/bookstore.git', branch: 'main', credentialsId: 'gitea-ci'
                    dir('bookstore-web') {
                        sh 'node --version'
                        sh 'npm ci --no-audit --no-fund --loglevel=error'
                        sh 'npx ng test --watch=false'
                    }
                }
            }
        }
    }
}

[Pipeline] { (Branch: Matrix - NODE_VERSION = '22')
[Pipeline] { (Branch: Matrix - NODE_VERSION = '24')
error: no such object: node:22-trixie
+ node --version
v24.22.2
 Test Files  2 passed (2)
      Tests  4 passed (4)
+ node --version
v22.23.4
 Test Files  2 passed (2)
      Tests  4 passed (4)
Finished: SUCCESS

The error: no such object line is not an error: Docker Pipeline checks with docker inspect whether the image is available locally and pulls it if not. With two axes (for example Node version and browser), matrix generates every combination; you remove the combinations you do not want with excludes. The price of matrix is executors: every cell needs its own agent.

Deploying to Windows and IIS

In corporate environments the target is often Windows and IIS. The flow is the same (take the package, stop the app, replace the files, start, health check), but the tools differ: an agent labeled windows, the powershell step and IIS's app_offline.htm mechanism. When the ASP.NET Core Module sees an app_offline.htm file in the site root, it stops the application and releases the locks on the DLLs (Dynamic-Link Library); once you copy the files and delete app_offline.htm, the application starts with the new version. Since this machine has no Windows, the stage below was not run; it only shows the structure:

stage('Deploy to IIS') {
    when { branch 'main' }
    agent { label 'windows && iis' }
    steps {
        unstash 'package'
        powershell '''
            $site = 'C:\\inetpub\\bookstore'
            Set-Content -Path "$site\\app_offline.htm" -Value 'Deploying Bookstore'
            tar -xzf "bookstore-$env:VERSION.tgz" -C $site
            Remove-Item "$site\\app_offline.htm"
            Invoke-WebRequest -UseBasicParsing http://localhost/health
        '''
    }
}

Web Deploy (msdeploy) is a more advanced alternative. We will cover the IIS side in detail, the locked file problems and Application Initialization, in the IIS posts of the upcoming Web Servers and Hosting series.

In Production

An approval is not a substitute for the deploy. input records a person saying "yes"; but that person must be able to see what they are saying yes to (which version, how long it was tested on staging). Show the version in the message, leave the staging result on the build page.

Expose the deploy credential in the narrowest place: sshagent only around the deploy script. On the server, the deploy user's rights are narrow too: writing files and restarting a single service.

Try the rollback. Old versions stay in the releases/ folder; have the script that switches the symlink back, and a Jenkins job that runs it, ready before you need them. Do not forget to clean up old versions after a certain number either. We will discuss strategies such as blue-green and canary in the upcoming Deployment Strategies series.

Common Pitfalls

Holding an executor while waiting for approval. Calling the input step inside a stage with an agent locks the executor during the wait; use the declarative input directive or a stage without an agent.

Assuming the lock guarantees ordering. As you saw above, an old build can deploy after a new one; without milestone, the old version can end up on staging.

Rebuilding during the deploy. The package going to staging and the package going to production end up different.

Deploying without a health check. Copying files does not mean the application started; you count a deploy that returns 502 on the first request as "successful".

An input without a timeout. Unapproved builds wait for months, each holding a build record and sometimes a lock.

A Small Experiment

★☆☆ Add an approval stage without an agent to your own pipeline, set submitter to someone other than your own user, and try approving with your own user. What happens?

★★☆ Run the lock example with three concurrent builds, first without milestone, then with inversePrecedence and milestone; write down in a table which build deployed in which order.

★★★ Write a script named ci/rollback.sh: it should find the previous version under releases/, point current at it, restart the service and run the health check. Then wire it to a parameterized Jenkins job.

The Jenkinsfile has grown: API, Web, packaging, deploys to two environments and an approval. If other Bookstore applications need the same steps, you will want to share this code rather than copy it. That is the topic of the next post: Shared Libraries, vars/ and src/.