JMeter: Where It Is Still the Right Tool, Test Plans, Thread Groups, CLI Mode and the HTML Dashboard

Version note: The code in this post was built and run with .NET 10; the load tests were run with Apache JMeter 5.6.3 (on Java 21) against Bookstore.Api and SQL Server 2025 on Docker Compose.

Guess First

On the institutional side of Bookstore, the online bookstore used as the shared example across this site's series, Hakan's team is talking to a QA team that has worked with JMeter test plans for schools and libraries for years. Saying "switch to k6" is easy; but are there cases where JMeter is still the right tool? There are two technical questions too. If you put a 2-5 second timer at the thread group level of a test plan, is that wait applied once per session or before every request? JMeter's classic Thread Group is a closed model; what is the JMeter version of the problem we saw in post 01, and how do you avoid it? In this post we build the Bookstore scenario in JMeter, run it from the CLI and read the HTML dashboard.

When Is JMeter Still the Right Tool?

JMeter is the Apache Software Foundation's load-testing tool, written in Java and developed since 1998. Test plans are stored as XML (.jmx) files and are usually designed in a GUI. Compared with k6, its strengths are clear. The first is protocol support: without installing plugins, JMeter can send requests directly to a database over JDBC, to message brokers over JMS, and over protocols such as LDAP, FTP, SMTP and TCP. If you need to measure the capacity of a message queue or an LDAP directory, JMeter has a ready-made sampler. The second is existing investment: hundreds of .jmx files accumulated over the years, and a team that knows them, raise the cost of moving to a new tool. The third is test teams that do not write code: building scenarios in a GUI is more approachable for a QA engineer who does not know JavaScript.

The cases for k6 are clear too: test code living in the same repository as the application code and going through code review; running in CI as a lightweight binary; the open model being the default and thresholds being defined inside the test. For Bookstore's HTTP API we prefer k6. If the institutional side ever needs to load the reporting database directly over JDBC, we go back to JMeter.

The release status is this: the latest stable release is 5.6.3, published in January 2024. It runs on Java 8 and later, with Java 17 or later recommended. The project's next major release, 6.0, aims to raise the minimum Java version to 17 and is in development. At the time of writing there is no published 6.0 release. I used 5.6.3 with Java 21.

Anatomy of a Test Plan

A JMeter test plan is a tree, and an element's effect depends on where it sits in the tree. The Bookstore plan consists of these parts: HTTP Request Defaults holding the host and port of all requests in one place, a Header Manager for the Content-Type header, a CSV Data Set Config that reads the ISBNs, and a thread group that produces sessions. Inside the thread group there are samplers (HTTP requests), assertions and extractors under them, timers between them, and a controller. The CSV and the thread group look like this:

      <CSVDataSet guiclass="TestBeanGUI" testclass="CSVDataSet" testname="ISBNs">
        <stringProp name="filename">../data/isbns.txt</stringProp>
        <stringProp name="fileEncoding">UTF-8</stringProp>
        <stringProp name="variableNames">isbn</stringProp>
        <boolProp name="ignoreFirstLine">false</boolProp>
        <stringProp name="delimiter">,</stringProp>
        <boolProp name="quotedData">false</boolProp>
        <boolProp name="recycle">true</boolProp>
        <boolProp name="stopThread">false</boolProp>
        <stringProp name="shareMode">shareMode.all</stringProp>
      </CSVDataSet>
      <OpenModelThreadGroup guiclass="OpenModelThreadGroupGui" testclass="OpenModelThreadGroup" testname="Sessions (open model)">
        <stringProp name="OpenModelThreadGroup.schedule">rate(${__P(rate,10)}/sec) random_arrivals(${__P(duration,60)} sec)</stringProp>
        <stringProp name="OpenModelThreadGroup.random_seed"></stringProp>
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
      </OpenModelThreadGroup>

The CSV Data Set Config reads the isbns.txt we produced in post 02 line by line and puts the next value into the isbn variable on every iteration. shareMode.all means all threads share the same file cursor. That is sequential reading, not the Zipf-like distribution we used in k6. If you want a popularity distribution, you need to write the selection in Groovy in a JSR223 PreProcessor. Expressions such as ${__P(host,localhost)} are JMeter properties: they are given on the command line with -Jhost=..., and the default is used if they are not. That way the same plan runs unchanged in different environments.

Instead of the classic Thread Group I used the Open Model Thread Group that came with 5.5. The schedule expression rate(10/sec) random_arrivals(120 sec) starts 10 new sessions per second on average and spreads their arrivals randomly. Each session runs the tree once and ends. This corresponds to k6's constant-arrival-rate. The JMeter documentation still marks this thread group as experimental. In older versions, or for more flexible profiles, plugins such as the Concurrency Thread Group and Arrivals Thread Group from jmeter-plugins.org are used.

Timers and controllers:

        <UniformRandomTimer guiclass="UniformRandomTimerGui" testclass="UniformRandomTimer" testname="Think 2-5 s">
          <stringProp name="ConstantTimer.delay">2000</stringProp>
          <stringProp name="RandomTimer.range">3000</stringProp>
        </UniformRandomTimer>
        <ThroughputController guiclass="ThroughputControllerGui" testclass="ThroughputController" testname="10% checkout">
          <intProp name="ThroughputController.style">1</intProp>
          <boolProp name="ThroughputController.perThread">false</boolProp>
          <intProp name="ThroughputController.maxThroughput">1</intProp>
          <FloatProperty>
            <name>ThroughputController.percentThroughput</name>
            <value>10.0</value>
          </FloatProperty>
        </ThroughputController>

The answer to the first guess question lies in JMeter's scope rule. A timer runs before every sampler at its level and below. This Uniform Random Timer at the thread group level does not wait once per session but 2-5 seconds before every request. In the Bookstore scenario that is the behavior I want, but putting a timer at the wrong level and not noticing that the test produces far less load than expected is a common mistake. The Throughput Controller runs the samplers inside it in 10 percent of sessions; that is why the checkout flow is present in one out of ten sessions.

In the checkout flow a value returned by one request has to be carried into the next. The JSON Extractor writes the id from the cart creation response into the cartId variable. A Response Assertion checks the status code:

            <JSONPostProcessor guiclass="JSONPostProcessorGui" testclass="JSONPostProcessor" testname="cartId">
              <stringProp name="JSONPostProcessor.referenceNames">cartId</stringProp>
              <stringProp name="JSONPostProcessor.jsonPathExprs">$.id</stringProp>
              <stringProp name="JSONPostProcessor.match_numbers">1</stringProp>
              <stringProp name="JSONPostProcessor.defaultValues">NOT_FOUND</stringProp>
            </JSONPostProcessor>
          <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="PUT /api/carts/{id}/items/{isbn}">
            <stringProp name="HTTPSampler.path">/api/carts/${cartId}/items/${isbn}</stringProp>
            <stringProp name="HTTPSampler.method">PUT</stringProp>
            <boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
            <elementProp name="HTTPsampler.Arguments" elementType="Arguments">
              <collectionProp name="Arguments.arguments">
                <elementProp name="" elementType="HTTPArgument">
                  <boolProp name="HTTPArgument.always_encode">false</boolProp>
                  <stringProp name="Argument.value">{ "quantity": 1 }</stringProp>
                  <stringProp name="Argument.metadata">=</stringProp>
                </elementProp>
              </collectionProp>
            </elementProp>
          </HTTPSamplerProxy>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Status 201">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="49587">201</stringProp>
              </collectionProp>
              <stringProp name="Assertion.test_field">Assertion.response_code</stringProp>
              <intProp name="Assertion.test_type">8</intProp>
            </ResponseAssertion>

If the JSON Extractor finds no match it writes NOT_FOUND; the next request goes to /api/carts/NOT_FOUND/items/... and gets a 404. That is why every critical request in the chain needs an assertion. A sample whose assertion fails is counted as an error in the results.

GUI for Design, CLI for Running

JMeter's GUI is for building the plan and debugging with a single thread. A load test is not run from the GUI: the GUI, and especially listeners such as View Results Tree, keep every result in memory and draw it on screen; the CPU and memory of the load-generating machine go to the interface instead of the results. The best practices section of the JMeter documentation also recommends running load in CLI mode with listeners switched off. The run looks like this:

# PowerShell (jmeter.bat on Windows)
jmeter -n -t load/jmeter/bookstore.jmx -Jrate=10 -Jduration=120 `
    -l results/results.jtl -j results/jmeter.log -e -o results/report

-n is non-GUI mode, -t the test plan, -l the results file where every sample is written (JTL), -j JMeter's own log file, and -e -o generate the HTML dashboard when the test ends. Make a habit of passing -j: without it, jmeter.log lands in the folder you run the command from, usually the repository root. During the run JMeter prints a summary every 30 seconds:

summary +    696 in 00:00:30 =   23.1/s Avg:    13 Min:     1 Max:    55 Err:     0 (0.00%) Active: 76 Started: 603 Finished: 527
summary =   1247 in 00:00:59 =   21.0/s Avg:    14 Min:     1 Max:   126 Err:     0 (0.00%)
summary +    703 in 00:00:30 =   23.5/s Avg:    14 Min:     1 Max:   138 Err:     0 (0.00%) Active: 76 Started: 906 Finished: 830
summary =   1950 in 00:01:29 =   21.8/s Avg:    14 Min:     1 Max:   138 Err:     0 (0.00%)
summary +    670 in 00:00:30 =   22.4/s Avg:    13 Min:     1 Max:   132 Err:     0 (0.00%) Active: 71 Started: 1196 Finished: 1125
summary =   2620 in 00:01:59 =   22.0/s Avg:    14 Min:     1 Max:   138 Err:     0 (0.00%)
summary +      6 in 00:00:01 =    9.4/s Avg:    13 Min:     3 Max:    42 Err:     0 (0.00%) Active: 0 Started: 1200 Finished: 1200
summary =   2626 in 00:02:00 =   21.9/s Avg:    14 Min:     1 Max:   138 Err:     0 (0.00%)

The + lines show the last 30 seconds, the = lines the total since the start. 1,200 sessions started in 120 seconds, exactly 10 per second. The Active column is the number of sessions open at the same time: 71-76. Let us check with Little's Law: a session sends 2.3 requests on average and waits 3.5 seconds on average before each one, so it lasts about 8 seconds. With 10 sessions per second, about 80 sessions are expected at once; the measured 71-76 is close to that. There is one more detail in the last lines: when the schedule ends, JMeter also stops sessions that have not finished. That is why the list request was sent 1,166 times for 1,200 sessions. If you want the last sessions to complete, add a pause at the end of the schedule.

Reading the HTML Dashboard

When you open index.html in the folder produced with -e -o, the first page has the APDEX table, the request summary and the statistics table. The Charts menu holds time series: response time over time, requests per second, active threads and the percentile distribution. The same statistics are also in the statistics.json file:

Watch three things when reading the table. The percentile columns are called "90th pct", "95th pct" and "99th pct" in the dashboard and pct1ResTime, pct2ResTime, pct3ResTime in the JSON. Which percentiles are computed changes with the aggregate_rpt_pct1..3 properties. JMeter records times as whole milliseconds; for requests under 1 ms it lacks k6's microsecond resolution. The "Total" row puts different endpoints into a single distribution; as we said about k6's general line in post 03, the real information is in the endpoint rows. The order p95 is 37 ms here because the payment stub was not running in this run.

APDEX's default thresholds are 500 ms (satisfied) and 1,500 ms (tolerated). If Bookstore's SLO sets tighter targets such as 100 ms for book details, adjust these thresholds to the SLO with the jmeter.reportgenerator.apdex_satisfied_threshold and apdex_tolerated_threshold properties; otherwise everything looks green. Finally, JMeter has no direct equivalent of k6's thresholds. A CLI run ends with exit code 0 even when assertions fail. I tried it by pointing the plan at a closed port: the summary showed Err: 4 (100.00%) and the exit code was still 0. If you want a gate in CI, you have to write the limit check over the JTL or statistics.json in a separate step.

Production Notes

Check the heap setting. JMeter's startup script opens with a 1 GB heap by default (-Xms1g -Xmx1g). Thousands of threads, large responses or listeners that keep results in memory fill that quickly, and GC pauses show up in the results as latency of the load generator. You can enlarge the heap with the HEAP environment variable, but first turn off listeners and unnecessary response saving. Writing only the needed fields to the JTL (the jmeter.save.saveservice.* properties) reduces disk and memory use.

Watch the load generator itself. In this run JMeter spent only about 7 seconds of CPU time over three minutes, because the load was low. When the thread count climbs into the thousands, remember that the JVM runs each thread as a separate operating system thread; memory and context-switch costs are higher than with k6's goroutines. The numbers in this post were taken on a single machine and their absolute values are machine-specific.

The answer to the second guess question: the classic Thread Group works with a fixed number of threads; each thread waits for the response, then sends the next request. That is the closed model from post 01, and when the system slows down JMeter, like k6, reduces the load on its own. Slow requests dropping out of the measurement is known in the literature as coordinated omission. To model traffic arriving from outside, use rate-oriented components such as the Open Model Thread Group or the Constant Throughput Timer, and watch whether the "active threads" count matches the target rate. We measure how coordinated omission distorts percentiles in the next post.

Common Pitfalls

• Running load from the GUI with View Results Tree open.
• Putting a timer at the wrong level and not noticing it waits before every request.
• Carrying the JSON Extractor's NOT_FOUND value into the next request without an assertion.
• Treating the CLI run's exit code as a gate; assertion failures do not change the exit code.
• Leaving the APDEX thresholds at their defaults and settling for a score looser than the SLO.
• Modeling traffic from outside with the classic Thread Group and letting slowdowns drop out of the measurement.
• Running without -j and -l and leaving log and result files inside the repository.

A Small Experiment

★☆☆ Move the timer from the thread group level to sit only under the second sampler and run with the same schedule. How did the Active column change? Explain with Little's Law.

★★☆ Set apdex_satisfied_threshold to 100 ms and apdex_tolerated_threshold to 300 ms and regenerate the dashboard with jmeter -g results.jtl -o report2. Which endpoint fell into the "frustrated" zone?

★★★ Add a JDBC Connection Configuration and a JDBC Request to the plan and run the catalog search query directly against BookstoreDb. How much of the search time through the API is spent in the database?

We have learned to produce numbers with k6 and JMeter. In the next post we move on to reading them correctly: why the average misleads, why percentiles cannot be averaged, measuring coordinated omission with a real pause, the throughput-latency curve and finding the bottleneck with the USE method.