Jenkins Credentials: withCredentials, Credentials Binding and the Groovy String Interpolation Trap

Version note: The examples in this post were run with Jenkins 2.580.2 LTS (Long-Term Support), the Credentials Binding plugin 728, the SSH (Secure Shell) Agent plugin 433 and Gitea 28. The secret values in the examples are fake values generated only for this experiment.

Guess First

In a pipeline you bind a NuGet API (Application Programming Interface) key to an environment variable with credentials(), and Jenkins masks it in the log as ****. A developer says "it is masked, so it is safe" and adds these three lines: echo "$NUGET_API_KEY", echo "$NUGET_API_KEY" | base64 and echo "$NUGET_API_KEY" | cut -c1-6. Which of them leaks something of the secret into the log? And another one: if you see no difference in the log between sh 'echo $KEY' and sh "echo ${KEY}", are they equally safe?

Credential Types, Stores and Scopes

In Jenkins a credential is a secret the pipeline needs, stored encrypted on the controller. The most common types: Secret text (API key, token), Username with password, SSH Username with private key, Secret file (such as a kubeconfig or .npmrc) and Certificate. Credentials live in a store: Jenkins's own (system) store, a folder's store or a user's personal store.

The scope says who can use a credential. A credential in SYSTEM scope is used only by Jenkins itself: starting agents, connecting to the mail server and so on. A credential in GLOBAL scope can also be used by jobs. On the controller in this series, credentials are defined with JCasC (Jenkins Configuration as Code) and the values are read from files:

credentials:
  system:
    domainCredentials:
      - credentials:
          - basicSSHUserPrivateKey:
              scope: SYSTEM
              id: agent-ssh-key
              username: jenkins
              description: "SSH key the controller uses to start ssh-agent-1"
              privateKeySource:
                directEntry:
                  privateKey: "${readFile:/run/secrets/agent_ssh_key}"
          - usernamePassword:
              scope: GLOBAL
              id: gitea-ci
              username: jenkins-bot
              password: "${readFile:/run/secrets/gitea_token}"
              description: "Gitea bot token for checkout and commit status"
          - string:
              scope: GLOBAL
              id: nuget-api-key
              secret: "${readFile:/run/secrets/nuget_api_key}"
              description: "NuGet feed API key (demo value)"
          - basicSSHUserPrivateKey:
              scope: GLOBAL
              id: deploy-ssh-key
              username: deploy
              description: "SSH key for deploying to app servers"
              privateKeySource:
                directEntry:
                  privateKey: "${readFile:/run/secrets/deploy_ssh_key}"

${readFile:...} is one of JCasC's secret sources: it reads the value from a file mounted read-only into the container, so the YAML (YAML Ain't Markup Language) itself contains no secret. On the controller disk the value is stored encrypted:

          <id>nuget-api-key</id>
          <description>NuGet feed API key (demo value)</description>
          <secret>{AQAAABAAAAAwPyA/d2M4MB8tZ6HG/QUh4+vjertSrKkpZChD8n/9krvXVPuIwhduf6TFv0VJePKuwmGHunh/zUVgsP/AXRqhLQ==}</secret>

The encryption key sits in the secrets/ folder of the same JENKINS_HOME (master.key and hudson.util.Secret). So anyone who gets hold of the whole JENKINS_HOME can decrypt the secrets. That means backups must be protected like secrets too; we will come back to this in the eleventh post.

environment with credentials()

The shortest usage is binding the credential to a variable in the environment block:

pipeline {
    agent { label 'linux' }
    environment {
        NUGET_API_KEY = credentials('nuget-api-key')
    }
    stages {
        stage('Masked') {
            steps {
                sh 'echo "key is $NUGET_API_KEY"'
            }
        }
        stage('Interpolated') {
            steps {
                sh "echo key is ${NUGET_API_KEY}"
            }
        }
        stage('Masking limits') {
            steps {
                sh 'echo "$NUGET_API_KEY" | base64'
                sh 'echo "$NUGET_API_KEY" | cut -c1-6'
            }
        }
    }
}

The real output:

Masking supported pattern matches of $NUGET_API_KEY
+ echo key is ****
key is ****
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
		 Affected argument(s) used the following variable(s): [NUGET_API_KEY]
		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ echo key is ****
key is ****
+ + base64
echo ****
****Cg==
+ + echo ****
cut -c1-6
oy2dem

There are three interesting things. For Secret text, credentials() puts the value itself into the variable; for the Username with password type, NUGET_API_KEY becomes user:password, and the variables NUGET_API_KEY_USR and NUGET_API_KEY_PSW are defined as well. Second, in the Interpolated stage the log still shows ****, but Jenkins warns explicitly. Third, the limit of masking: it caught the base64 form too (the ****Cg== in the output; Cg== is the base64 tail of the trailing newline), but no mask recognizes the first six characters, and oy2dem landed in the log.

The Groovy String Interpolation Trap

If the two lines look identical in the log, why the warning? Because masking only protects the log output, not the process. When you write sh "echo key is ${NUGET_API_KEY}", Groovy inserts the value; the command line sent to the agent contains the secret in clear text. While that command runs, any process on the same machine can see the arguments with ps, the command is written as a shell script file into the @tmp folder, and on failure, exception messages created on the Groovy side can carry the value too. When you write it with single quotes, Groovy never sees the value: the command line contains only the text $NUGET_API_KEY, and the shell reads the value from the environment.

The rule is short: in every sh, bat or powershell step that involves a secret, use single quotes (or ''') and read the secret as a shell variable. If you need to combine another Groovy variable, put it into the environment with withEnv and still use single quotes. Never dismiss Jenkins's "A secret was passed to sh using Groovy String interpolation" warning as "just a warning"; if that line shows up in the log, there is a leak path in the pipeline.

Narrow Scope with withCredentials

A credential in the environment block stays exposed for the whole pipeline. To expose the secret only to the few lines that need it, you use withCredentials. A stage that calls the Gitea API with the bot user's token:

stage('Call Gitea API') {
    steps {
        withCredentials([usernamePassword(credentialsId: 'gitea-ci',
                                          usernameVariable: 'GITEA_USER',
                                          passwordVariable: 'GITEA_TOKEN')]) {
            sh '''
                curl -s -H "Authorization: token $GITEA_TOKEN" \
                    http://jenkwork-gitea:3000/api/v1/user | grep -o '"login":"[^"]*"'
            '''
        }
    }
}

Masking supported pattern matches of $GITEA_TOKEN
+ curl -s -H Authorization: token **** http://jenkwork-gitea:3000/api/v1/user
+ grep -o "login":"[^"]*"
"login":"jenkins-bot"

The token exists only inside this block and is removed from the environment when the block ends. The binding types match the credential types: string, usernamePassword, file (writes the secret file to a temporary path and puts the path into the variable), sshUserPrivateKey and certificate.

If you connect to a server over SSH, the sshagent step loads the private key into an ssh-agent process without writing it to disk:

sshagent(credentials: ['deploy-ssh-key']) {
    sh 'ssh-add -l'
}

[ssh-agent] Using credentials deploy (SSH key for deploying to app servers)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-DJAj47Qe3YV3/agent.640
SSH_AGENT_PID=643
Running ssh-add (command line suppressed)
[ssh-agent] Started.
+ ssh-add -l
256 SHA256:sRV1aiVVwjruTv9kTull+IRG64Db4ux1/IwYvXmUa5c jenkins-deploy (ED25519)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 643 killed;
[ssh-agent] Stopped.

The ssh, scp and rsync commands inside the block use the key through SSH_AUTH_SOCK, and the agent is killed when the block ends. We will use this in the deploy in the ninth post.

The Real Effect of Scope: SYSTEM and Folders

I added a stage to the same pipeline that uses agent-ssh-key, which is in SYSTEM scope:

stage('System scope') {
    steps {
        withCredentials([sshUserPrivateKey(credentialsId: 'agent-ssh-key', keyFileVariable: 'KEY')]) {
            sh 'ls -l "$KEY"'
        }
    }
}

ERROR: Could not find credentials entry with ID 'agent-ssh-key'
Finished: FAILURE

The credential exists, but not for jobs. The key that starts agents being unreadable from a pipeline is exactly what we want; otherwise any Jenkinsfile could send that key to another server.

The second tool for team isolation is folder-level credentials. I added a web-deploy-token to a folder named team-web that only jobs in that folder can see (with Job DSL (Domain-Specific Language), through the folder's folderCredentialsProperty). I ran the same Jenkinsfile first inside the folder, then at the root:

withCredentials([string(credentialsId: 'web-deploy-token', variable: 'DEPLOY_TOKEN')]) {
    sh 'echo "token length: ${#DEPLOY_TOKEN}"'
}

The team-web/web-deploy job inside the folder:

Running on ssh-agent-1 in /home/jenkins/agent/workspace/team-web/web-deploy
Masking supported pattern matches of $DEPLOY_TOKEN
+ echo token length: 23
token length: 23
Finished: SUCCESS

The api-deploy job at the root:

Running on docker-agent-1 in /home/jenkins/agent/workspace/api-deploy
ERROR: Could not find credentials entry with ID 'web-deploy-token'
Finished: FAILURE

Combined with authorization, folders become a strong boundary: the web team manages the credentials in its own folder, and the API team's jobs cannot see them. The second benefit of folder-scoped credentials is avoiding name collisions; both teams can have their own deploy-token.

External Secret Stores

Keeping credentials in Jenkins's own store is enough for small and medium installations. If the organization already has HashiCorp Vault or Azure Key Vault, the corresponding plugins read the secret from there at build time and only a reference stays in Jenkins. The benefit is rotation and auditing in one place; the price is one more plugin and one more dependency. We will discuss these stores themselves in the upcoming Configuration and Secret Management series; the rules in this post (narrow scope, single quotes, not trusting masking) are the same whichever store you use.

In Production

Define credentials in the narrowest scope: agent and infrastructure keys in SYSTEM, team secrets in folders, and only the truly shared ones global. Give every credential a description saying what it is for and who owns it; two years later nobody can answer "what was deploy2-new?".

Do not use production credentials in pull request builds. If a PR (Pull Request) from a fork can change the Jenkinsfile, it can add curl attacker.example inside withCredentials. In the sixth post we will see how fork PRs are prevented from changing the Jenkinsfile; still, put deploy credentials only in stages that run on the main branch.

Plan rotation. If a token stored in Jenkins has no expiry date, there is also no mechanism that would notice when it leaks. Short-lived tokens (GitHub App tokens, temporary credentials issued by cloud providers through OIDC (OpenID Connect)) are better than static secrets wherever possible.

Common Pitfalls

Treating masking as a security boundary. Masking catches only the full value and some known encodings; a fragment, the reversed value, the URL (Uniform Resource Locator) encoded value or a copy written to a file and archived is not caught. Anyone who can write the pipeline can read the secret: masking is a net against accidental leaks, not against deliberate reading.

Interpolation with double quotes. Do not let **** in the log fool you; if the warning line is there, there is a problem.

set -x and debug output. The shell's xtrace is masked, but tools' own debug logs (for example the headers printed by curl -v or a CLI's (Command Line Interface) verbose mode) can print the secret in a different form.

Putting a secret into an artifact or a Docker image. A connection string written into an appsettings.Production.json during the build is copied to the controller with archiveArtifacts, and anyone who downloads it can read it.

Not encoding the environment in credential IDs. Names like bookstore-db-staging and bookstore-db-production instead of db-password make the mistake of deploying to the wrong environment more visible.

A Small Experiment

★☆☆ Create a Secret text credential, bind it in environment with credentials(), and print it with both a single-quoted and a double-quoted sh. Find the warning line in the log.

★★☆ Bind a Username with password credential with credentials() and observe what the _USR and _PSW variables contain and what the main variable turns into (without printing the values, showing their lengths with ${#VAR}).

★★★ Create two folders, put a credential with the same ID but a different value into each, and also define a global credential with the same ID at the root. Which one does a job inside a folder see? Find the credential lookup order by experiment.

In the next post we connect Jenkins to the repository: multibranch pipelines, webhooks, pull request builds, trust settings for fork PRs and the status check sent back to the commit.