jq and yq in Depth: Querying, Transforming and Editing JSON and YAML on the Command Line

Version note: The examples in this post were run with jq 1.8.2, yq (mikefarah) 4.54.3, .NET 10 (SDK 10.0.404), PowerShell 7.7, kubectl 1.37 (against a local kind cluster), curl and actionlint 1.7.12. For comparison, yq 3.4.1 (container image) and the Python yq package 4.4.5 were also tried.

Guess First

We wrote two commands that fetch the paid orders from the API of Bookstore, the online bookstore used as the shared example across this site's series. A CI script will look at this output and stop if there are no paid orders:

$ curl -s http://127.0.0.1:52606/api/orders | jq -r '.[] | select(.status == "paid") | .orderNumber'
$ curl -s http://127.0.0.1:52606/api/orders | jq -r '.[].ordernumber'

What do both print, and what are their exit codes? Can the script tell from the exit code that there are no paid orders?

The Filter Chain: .[], select, map

jq was introduced in the modern CLI tools post of the Terminal and Command Line series; here we go deeper. The examples run against a version of Bookstore.Api slimmed down for this series, working with in-memory data instead of a database. Start the API in a separate terminal:

$ dotnet run --project src/Bookstore.Api --urls http://127.0.0.1:52606

Everything in jq is a filter: it takes an input and produces zero or more outputs; | hands one filter's output to the next. .[] streams the elements of an array one by one, select(condition) drops the ones that don't satisfy the condition, and map(f) is short for [.[] | f], collecting the result back into an array. The curl side is in the curl post of the API Tools series:

$ curl -s http://127.0.0.1:52606/api/books | jq '.[0]'
{
  "isbn": "9780441013593",
  "title": "Dune",
  "author": "Frank Herbert",
  "price": 24.90,
  "stock": 12
}
$ curl -s http://127.0.0.1:52606/api/books | jq -c 'map(select(.stock > 0) | {isbn, title})'
[{"isbn":"9780441013593","title":"Dune"},{"isbn":"9780547928227","title":"The Hobbit"},{"isbn":"9780141439549","title":"Middlemarch"}]
$ curl -s http://127.0.0.1:52606/api/books | jq '[.[] | .price * .stock] | add'
489.8499999999999

The last line recalls the rule from the first post of this series: jq 1.8 keeps the original text of a number, but uses doubles as soon as you do arithmetic. Don't do money calculations such as stock value in jq; even for a report you would have to round the result.

null, Empty Output and Exit Codes with -e

The answer to the guess:

$ curl -s http://127.0.0.1:52606/api/orders | jq -r '.[] | select(.status == "paid") | .orderNumber'; echo "exit: $?"
exit: 0
$ curl -s http://127.0.0.1:52606/api/orders | jq -r '.[].ordernumber'; echo "exit: $?"
null
null
null
exit: 0
$ curl -s http://127.0.0.1:52606/api/orders | jq -e -r '.[] | select(.status == "paid") | .orderNumber'; echo "exit: $?"
exit: 4
$ curl -s http://127.0.0.1:52606/api/orders | jq -e -r '.[].ordernumber'; echo "exit: $?"
null
null
null
exit: 1

The first command printed nothing, because the status value is "Paid", not "paid"; the comparison is case-sensitive. The second printed null three times because of the typo in the field name, since jq returns a missing field as null, not as an error. Both exit with 0; the script cannot tell that something is wrong. -e (--exit-status) ties the exit code to the result: 1 if the last output is null or false, 4 if there was no output at all. Add -e to every jq command you use as a condition in CI. When you want to handle a missing value on purpose, the // operator helps: .status // "Unknown" returns the right side if the left side is null or false.

group_by, reduce, --arg and --argjson

Order totals and grouping by status:

$ curl -s http://127.0.0.1:52606/api/orders | jq -c '.[] | {orderNumber, total: ([.items[] | .quantity * .unitPrice] | add)}'
{"orderNumber":"BS-1001","total":49.8}
{"orderNumber":"BS-1002","total":37.15}
{"orderNumber":"BS-1003","total":560.25}
$ curl -s http://127.0.0.1:52606/api/orders | jq -c 'group_by(.status) | map({status: .[0].status, count: length})'
[{"status":"AwaitingApproval","count":1},{"status":"Paid","count":1},{"status":"Shipped","count":1}]

group_by(.status) sorts the array and splits elements with the same value into sub-arrays; the map after it reduces each group to a single object. There are two ways to pass values in from outside, and the difference matters:

$ curl -s http://127.0.0.1:52606/api/books | jq --arg min 15 'map(select(.price >= $min)) | length'
0
$ curl -s http://127.0.0.1:52606/api/books | jq --argjson min 15 'map(select(.price >= $min)) | length'
3
$ curl -s http://127.0.0.1:52606/api/books | jq --arg isbn 9780547928227 -r '.[] | select(.isbn == $isbn) | .title'
The Hobbit

--arg always passes the value as a string; since price is a number, the comparison 24.90 >= "15" is always false in jq's type ordering (numbers sort before strings) and the result is 0. --argjson parses the value as JSON and gives the right result. When comparing with a string field such as an ISBN, --arg is the right choice. Don't embed the value into the filter by string concatenation; you get quoting problems, and the value is interpreted as part of the filter.

Output Formats: -r, @csv, @tsv and Environment Variables

-r writes strings without quotes; @csv and @tsv turn an array into a line:

$ curl -s http://127.0.0.1:52606/api/books | jq -r '.[] | [.isbn, .title, .price] | @csv'
"9780441013593","Dune",24.90
"9780141439587","Emma",14.50
"9780547928227","The Hobbit",19.90
"9780141439549","Middlemarch",17.25
$ curl -s http://127.0.0.1:52606/api/books | jq -r '.[] | [.title, .stock] | @tsv'
Dune	12
Emma	0
The Hobbit	7
Middlemarch	3

@csv quoted the strings and left the numbers as they are; the trailing zero of 24.90 was kept too, because no arithmetic was done on the value. That is all a quick stock report for Bookstore's back office takes. A more practical example is producing environment variables .NET understands from the JSON settings file of the fourth post of this series:

$ jq -r 'paths(scalars) as $p | "\($p | map(tostring) | join("__"))=\(getpath($p))"' settings/shipping.json
Shipping__DefaultCountry=NO
Shipping__Cutoff=12:30
Shipping__Express=true
Shipping__MaxWeightGrams=1000
Shipping__ApiVersion=1.10
Shipping__Carriers__0=standard
Shipping__Carriers__1=express
$ jq -c '.Shipping | with_entries(select(.value | type != "array"))' settings/shipping.json
{"DefaultCountry":"NO","Cutoff":"12:30","Express":true,"MaxWeightGrams":1000,"ApiVersion":"1.10"}

paths(scalars) produces the path of every leaf value (such as ["Shipping", "Carriers", 0]), getpath reads the value at that path, and join("__") builds .NET's hierarchy separator. You can put the output into a Compose env_file or a Kubernetes manifest. with_entries works on the key-value pairs of an object; here it left out the fields that are arrays.

Large Files: inputs, -c and JSON Lines

--slurp (-s) collects all input into a single array; convenient for small files, but it loads the whole file into memory. For the JSON Lines files we covered in the first post of this series, a better way is -n with inputs: jq does not read the input itself, inputs yields the next value on each call, and reduce accumulates them one by one:

$ curl -s http://127.0.0.1:52606/api/orders | jq -c '.[]' > orders.jsonl
$ jq -n 'reduce inputs as $order (0; . + ($order.items | map(.quantity) | add))' orders.jsonl
29

Because -c writes each value on a single line, it turned the API response into JSON Lines; reduce inputs computed the total number of books across all orders without holding an array in memory. Instead of typing a long filter on the command line you can also put it in a file:

# Order totals per status; keeps working on Windows shells because nothing needs quoting
map({status, total: ([.items[] | .quantity * .unitPrice] | add)})
| group_by(.status)
| map({status: .[0].status, orders: length, total: (map(.total) | add)})

$ curl -s http://127.0.0.1:52606/api/orders | jq -f order-totals.jq
[
  {
    "status": "AwaitingApproval",
    "orders": 1,
    "total": 560.25
  },
  {
    "status": "Paid",
    "orders": 1,
    "total": 49.8
  },
  {
    "status": "Shipped",
    "orders": 1,
    "total": 37.15
  }
]

You can use comment lines in a filter file passed with -f, and most importantly it removes the shell-dependent quoting problem entirely.

Real Sources: kubectl, dotnet, the GitHub API and PowerShell

The real use of jq is the JSON output of other tools. We applied Bookstore's manifest to the local kind cluster; because the image is not in a registry, the pods don't come up. Let's find which pod is waiting and why, in one line:

$ kubectl apply -f deploy/k8s/api.yaml
configmap/api-config unchanged
deployment.apps/api unchanged
service/api unchanged
$ kubectl get pods -o json | jq -r '.items[] | [.metadata.name, (.status.containerStatuses[0].state | keys[0]), (.status.containerStatuses[0].state.waiting.reason // "-")] | @tsv'
api-79559dd97f-jlw5n	waiting	ImagePullBackOff
api-79559dd97f-wsf52	waiting	ImagePullBackOff
$ kubectl get deployment api -o json | jq -c '{desired: .spec.replicas, ready: (.status.readyReplicas // 0)}'
{"desired":2,"ready":0}

kubectl's own -o jsonpath output is enough for simple fields; when you need conditions, defaults (//) or formatting, jq is more comfortable. The same approach works with the .NET SDK's JSON output:

$ dotnet list src/Bookstore.Api package --outdated --format json | jq -r '.projects[].frameworks[]?.topLevelPackages[]? | "\(.id) \(.resolvedVersion) -> \(.latestVersion)"'
Microsoft.AspNetCore.OpenApi 10.0.15 -> 11.0.0

GitHub's REST API returns JSON too. A query that needs no authentication: the latest jq release and its Windows binaries:

$ curl -s https://api.github.com/repos/jqlang/jq/releases/latest | jq -r '.tag_name, (.assets[].name | select(test("windows")))'
jq-1.8.2
jq-windows-amd64.exe
jq-windows-arm64.exe
jq-windows-i386.exe

gh api does the same with authentication and runs jq syntax internally through its --jq option; jq does not need to be installed separately. The Azure CLI uses a different query language, JMESPath, through --query; you can also take az output with -o json and pipe it to jq. I did not run these two for this post.

On Windows, PowerShell itself is an alternative to jq: Invoke-RestMethod and ConvertFrom-Json turn JSON into objects, and filtering is done with Where-Object. There is one trap:

$ pwsh -NoProfile -Command '$books = Invoke-RestMethod http://127.0.0.1:52606/api/books; $books | Where-Object stock -eq 0 | Select-Object -ExpandProperty title'
Emma
$ pwsh -NoProfile -Command '$orders = Invoke-RestMethod http://127.0.0.1:52606/api/orders; ConvertTo-Json -InputObject $orders -Compress' | cut -c1-150
WARNING: Resulting JSON is truncated as serialization has exceeded the set depth of 2.
[{"orderNumber":"BS-1001","customerName":"Ada","status":"Paid","items":["@{isbn=9780441013593; quantity=2; unitPrice=24.9}"]},{"orderNumber":"BS-1002"

The default depth of ConvertTo-Json is 2. Because the items inside the orders sit on the third level, they turned into strings of the form @{isbn=...} instead of JSON, and only a warning was printed. When writing JSON back from PowerShell, always pass -Depth explicitly.

yq: Querying and Editing YAML in Place

mikefarah/yq runs a syntax very similar to jq on YAML, JSON, XML, TOML and INI. Reading and format conversion:

$ yq 'select(.kind == "Deployment") | .spec.template.spec.containers[0].image' deploy/k8s/api.yaml
bookstore-api:1.10.0
$ yq -o json -I0 'select(.kind == "Service") | .spec.ports' deploy/k8s/api.yaml
[{"name":"http","port":80,"targetPort":"http"}]
$ yq -p json -o yaml '.Shipping' ../labs/04-config/settings/shipping.json
DefaultCountry: NO
Cutoff: 12:30
Express: true
MaxWeightGrams: 1000
ApiVersion: "1.10"
Carriers:
  - standard
  - express

Watch the last command: when producing YAML from JSON, yq wrote NO and 12:30 without quotes, because both are strings in YAML 1.2. It quoted 1.10, because unquoted it would be a float. If one of the YAML 1.1 tools we saw in the second post of this series (PyYAML, kubectl) reads this output, Norway becomes false again. If you hand YAML produced by yq to another tool, check it against that tool's rules.

The most common job for yq is updating the image tag in a manifest in a release pipeline. Running it first without -i and checking what will change with diff is a good habit:

$ TAG=1.11.0 yq '(select(.kind == "Deployment") | .spec.template.spec.containers[] | select(.name == "api") | .image) = "bookstore-api:" + strenv(TAG)' deploy/k8s/api.yaml | diff deploy/k8s/api.yaml -
37c37
<           image: bookstore-api:1.10.0
---
>           image: bookstore-api:1.11.0
$ TAG=1.11.0 yq -i '(select(.kind == "Deployment") | .spec.template.spec.containers[] | select(.name == "api") | .image) = "bookstore-api:" + strenv(TAG)' deploy/k8s/api.yaml
$ grep -n -B1 "image:" deploy/k8s/api.yaml
36-          # the release pipeline updates this tag with yq
37:          image: bookstore-api:1.11.0

Only the image line changed. The --- separators of the multi-document file, the | block in the ConfigMap and the comment above the image stayed as they were; yq tries to preserve comments and formatting when editing. strenv(TAG) reads the value from the environment variable as a string, so a tag like 1.10 does not turn into a number. In a GitOps flow a pipeline makes this commit and Argo CD updates the cluster from the repo; that flow is the subject of the GitOps and Argo CD series.

Two Tools Named yq and Version Differences

There are two different tools called "yq", and even their version numbers look alike. The mikefarah/yq in this post is written in Go, a single binary. The yq package in Python (kislyuk/yq) is a wrapper that turns YAML into JSON and hands it to the real jq:

$ python -m venv .venv-pyyq && .venv-pyyq/bin/pip install -q --disable-pip-version-check yq==4.4.5
$ .venv-pyyq/bin/yq --version
yq 4.4.5
jq-1.8.2
$ yq --version
yq (https://github.com/mikefarah/yq/) version v4.54.3
$ .venv-pyyq/bin/yq -r 'select(.kind == "Deployment") | .spec.template.spec.containers[0].image' ../../bookstore/deploy/k8s/api.yaml
bookstore-api:1.11.0

Both introduce themselves as "yq 4". The Python yq uses jq syntax exactly but writes JSON by default (-y for YAML), requires jq to be installed and does not preserve comments. When a CI script says "yq", check which one runs with yq --version. mikefarah/yq has its own break too: in v3 commands were subcommands such as yq r and yq w with dot-separated paths; v4 brought jq-like expressions:

$ docker run --rm --platform linux/amd64 -v "$PWD":/workdir mikefarah/yq:3 yq r -d1 deploy/k8s/api.yaml 'spec.template.spec.containers[0].image'
bookstore-api:1.11.0
$ yq r -d1 deploy/k8s/api.yaml 'spec.template.spec.containers[0].image' 2>&1 | head -2
Error: unknown shorthand flag: 'd' in -d1
Usage:

Running an old example from the internet with v4 gives you an unhelpful flag error. Check the version when reading documentation; new operators are added within v4 as well, so pin the yq version in CI.

A Condition with jq in CI

In Bookstore's CI workflow, a step after the build stops the job if a NuGet dependency has a known vulnerability:

name: ci

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.14"
      - run: pip install yamllint==1.38.0 check-jsonschema==0.39.0 json5==0.17.3
      # -f github turns every finding into an annotation on the pull request
      - run: yamllint -f github .
      - name: Validate settings against Bookstore's schema
        run: >-
          check-jsonschema --force-filetype json5
          --schemafile schema/bookstore-settings.schema.json
          src/Bookstore.Api/appsettings*.json
      - name: Validate Compose and workflow files
        run: |
          check-jsonschema --builtin-schema vendor.compose-spec compose.yaml
          check-jsonschema --builtin-schema vendor.github-workflows .github/workflows/*.yml

  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-dotnet@v6
        with:
          dotnet-version: 10.0.x
      - name: Build
        run: |
          dotnet restore src/Bookstore.Api
          dotnet build src/Bookstore.Api -c Release --no-restore
      - name: Fail on vulnerable packages
        run: |
          dotnet list src/Bookstore.Api package --vulnerable --format json > packages.json
          # -e: exit code 1 when the result is false, so the step fails
          jq -e '[.projects[].frameworks[]?.topLevelPackages[]?] | length == 0' packages.json

We tried the step locally with the same commands:

$ dotnet list src/Bookstore.Api package --vulnerable --format json > packages.json
$ jq -e '[.projects[].frameworks[]?.topLevelPackages[]?] | length == 0' packages.json; echo "exit: $?"
true
exit: 0
$ actionlint .github/workflows/ci.yml && echo "actionlint: no findings"
actionlint: no findings
$ kubectl delete -f deploy/k8s/api.yaml
configmap "api-config" deleted from default namespace
deployment.apps "api" deleted from default namespace
service "api" deleted from default namespace

With no vulnerable package the result is true and the exit code 0. If a package shows up in the list, the expression becomes false, -e makes the exit code 1 and the step turns red. The question marks in [...]? matter: when there are no vulnerabilities, the --vulnerable output may have no frameworks field at all; without ? jq would fail with "cannot iterate over null". To carry a value from jq to the next step in GitHub Actions you write it to GITHUB_OUTPUT; that mechanism is in the expressions and contexts post of the GitHub Actions series.

In Production

Pin the jq and yq versions in scripts; in jq 1.8 the behavior of some functions such as tonumber and indices changed, and every yq release brings new operators. Use -e in every filter that acts as a condition in CI. In pipelines that edit in place (yq -i), write the change to the log with diff first; an unexpected change is noticed before it is committed. Always pass values from users into a filter with --arg, --argjson or strenv().

Common Pitfalls

Quoting on Windows is where people get stuck most. According to the jq manual, in cmd.exe you put the filter in double quotes and escape the double quotes inside with backslashes; in PowerShell you use single quotes and still escape the double quotes inside. The same command is written three different ways in bash, cmd and PowerShell; putting filters shared within the team into a file with -f ends the problem.

The second pitfall is null slipping through silently: a typo in a field name is not an error in jq but null, which with -r becomes the word null on screen and the string "null" in a shell variable. The third is yq version confusion: v3 examples, and mixing up Python yq with mikefarah yq. The fourth is that YAML produced by yq can be read differently by YAML 1.1 tools.

A Small Experiment

★☆☆ Fix the two commands from the guess: make the status comparison case-insensitive (ascii_downcase) and add -e. Which exit codes do you get?

★★☆ Add the average order amount per status to order-totals.jq and turn the result into a report with @csv. Which jq 1.8 functions can you use to round the amounts to two decimals?

★★★ Write a release script that updates both the image tag and the app.kubernetes.io/version label in deploy/k8s/api.yaml to the same version, shows the change with diff, and verifies with yq '... | tag' that the label is still a string.

The series ends here. In six posts we saw what JSON's specification leaves open, the differences between YAML's two versions, multiline strings and anchors, the rules of TOML, INI and .env that change from parser to parser, validation with JSON Schema, and jq and yq. The common lesson fits in one sentence: the meaning of a config file depends on the tool that reads it; validate with that tool, with a schema where possible, and in CI. The files in this series can serve as a checklist for every compose.yaml, workflow and manifest you write in Bookstore's DevOps series.