Intive-FDV Blog

Taking JMeter Tests to the Next Level with Taurus

They say that necessity is the mother of invention, and it was necessity that made us do a Google search and get to know Taurus. Taurus is a wrapper for JMeter tests, although the term wrapper falls a bit short. The need to write this article came up after we had to test a system with more than a 100 thousand requests per minute, which pushed the infrastructure to the limit and forced us to take our testing to the next level. Here I’m going to tell you what we learned.

The Beginning of the Story

JMeter is a basic tool that every developer should know. Once you master it, it allows to effectively execute stress tests and automate sanity tests. At first, JMeter can seem a bit unpolished, but when you start running the scripts in a computer, you start to appreciate its power. And when you start using it to create distributed tests from several servers at the same time, it’s magical.

Going back to our case, we considered using JMeter just to measure performance in relation to response times and http codes, we weren’t interested in adding assertions nor extracting data from the responses. This is a step-by-step description of what we did.

Creating an App for Testing

We’re going to use Node and Express to create a simple web server. In the terminal, we go to the folder where we want to save the code and we execute “npm init”. We complete the data requested and then run “npm install -D express”, which installs Express. Finally, we create a file named “app.js” with the following content (comments in the code explain each item):

 

There are three endpoints: one that always responds well, another one that is slow, and other one that fails intermittently.

We execute “node app.js” and we have the application up and running, ready to be tested.

Testing with JMeter

Now, with JMeter we are going to create a basic, configurable test so that we don’t need to go back. First, we add the element “setUp Thread Group”. There we define the default values for number of users (or threads), ramp-up period and loop count, but we can overwrite them each time we run the test. That will make our life easier later.

In Number of Threads we put “${__P(threads,1)}”, where threads is a parameter we can set when running the test. If empty, the default value is 1. In Ramp-Up Period we put “${__P(rampup,1)}”, and in Loop Count, “${__P(loops,1)}”. We will have something like this:

While tt’s not necessary for this tests, we could define, for example, an element in User-Defined Variables to have control over other test elements.

Then, we are going to add which endpoints calls we want to make. In HTTP Request we add three elements and fill in the information that corresponds to the Server (localhost), Port number (3000) and Path (fast, slow, or fail). For example:

Lastly, we add the listener Aggregate Report in order to see a summary of our testing. This listener will provide information such as average response time, median, 90, 95 and 99 percentiles, and error percentage. Percentiles are especially useful in high-load scenarios, because the average can show values that don’t match the users’ general experience.

The current configuration only helps to check that everything is working well, since it will just make one request at each endpoint. If we add more threads, it starts to get interesting. If we add a thousand threads, for example, we get something like this:

It’s a lot of information and it can be hard to understand. The Median, 90, 95 and 99 percentiles columns show the maximum time of the requests. For example, for the endpoint /fast, 90%: 272 means that 90% of the requests took 272 milliseconds or less.

We can observe that the slow endpoint is quite slow indeed: 50% of the requests took more than 582 miliseconds. We can also conclude that 40% of the requests made to that endpoint took between 582 and 971 milliseconds.

On the other hand, in the Error Percentage column we can see that 76% of the requests to /fail showed errors of some kind, which are all requests that didn’t get a http 200 code as response. That can be configured, but it wasn’t something we needed for our use case.

Running Tests from the Terminal

The graphic interface of JMeter is only designed to create tests. In order to leverage all their power, it’s better to run them from the terminal. We should save the test “test.jmx” and go to a console, where we can run the test with the command “jmeter -n -t test.jmx”.

The result may look uninteresting, given that we’re running the test with one thread, one second for ramp-up period and one loop count as default values. Let’s set these parameters to appreciate their power. Parameters can be set with -J<parameterName>=<value>. As an example, we can execute: “jmeter -n -t test.jmx -Jthreads=2000 -Jrampup=33”, which will run the tests we want with 2000 users, who will “connect” with each other in 33 seconds.

The results showing in the console may not be as informative as we want to, but I can assure you that they provide a lot of information, especially if we run load tests for a long time. With those parameters, we can get an output that looks like this:

If we want to generate an output, we can run the same test with the parameter “-l” to specify the file where we want to keep the testing information. So, for example, we could run “jmeter -n -t test.jmx -Jthreads=2000 -Jrampup=33 -l out.jtl” and then upload the file out.jtl in the graphic interface of JMeter in order to see the results with the listeners we added.

Reporting and Testing with Blazemeter

JMeter offers listeners that are useful to create reports about the tests, but they look a bit raw to show to end users or to people who don’t have a background in Systems. There are other tools that allow better visualizations, and some even allow to run tests from a web environment. Blazemeter is one of them.

Undoubtedly, reports are highly improved thanks to this tool, and the charts are very friendly to end users. The above image shows the overview, that is, the most basic view. In Timeline Report we can easily create reports about the most important KPIs.

The free version of Blazemeter allows to try different things and to generate tests that don’t involve high loads. For more complex stress tests, you may want to consider paying a more complete plan. But even the paid plans may fall short when you try to push the limits, like we did when we needed to generate 20 thousand requests per minute and we didn’t have the budget to get a custom plan.

Luckily, Blazemeter offers an open source tool called Taurus, thanks to which we can easily coordinate testing.

Trying Taurus

In order to use Taurus we need Python. I suggest creating the tests in a virtual environment. Assuming everything is already installed, it’s as simple as running “virtualenv Taurus” and then “source ./taurus/bin/activate”. Once the virtual environment is created, the next step is to install Taurus with “pip install bzt”.

The fastest way to check that everything is in order is to create a file test.yml with the following content:

 

After downloading a few additional dependencies (just to run it for the first time), it will try to keep 100 threads at the same time for 5 minutes, and it will display results in the console in a friendly way. When that happens, it means that everything is ready to begin testing with Taurus.

The results are quite interesting. Although at the beginning the interface can be intimidating, it provides a lot of useful information. Top down, from left to right, it displays the following:

  • 3 charts for:
    • Total and active users
    • Hits and errors
    • Response times
  • Statistics of the latest interval (sampling)
    • Response times
    • Percentiles
    • Response codes
  • Cumulative statistics using the same data
  • Main endpoints
  • Main errors

Lastly, below the Taurus logo, there are:

  • General test statistics
  • Status of the server where we are running the tests
  • Messages generated by Taurus

These tests contain errors, probably because I’m working in my computer and it isn’t able to process all the requests.

Executing JMeter with Taurus

The easiest way to do this is to save files .jmx and .yml in the same folder. I renamed the JMeter test “jmeter.jmx”, just out of preference. We keep the configuration file test.yml with the following content:

 

We’re telling it to run the test we defined in JMeter and to overwrite the number of threads and the loop count. Thanks to that, the test will run really quickly, and probably we won’t see the interface. So, everything is ready for testing in a very elegant way, we are just missing reporting features. It’s worth noting that Taurus creates a folder containing the output of JMeter, and with a little patience you could generate reports based on those raw data.

For example, it’s possible to add a reporting feature to compare executions. A quick way to do this is to use the final-stats module. The configuration will look as this:

 

I added iterations and concurrency to have a data volume that looks more interesting.

While the test is running, what we see is virtually identical to the previous executions, perhaps more information is displayed because we’re making a lot more requests.

When the test ends, we get a summary indicating percentiles, error percentage, test duration, fails per endpoint, etc.

All the data is saved in an xml file which we can then import into a tool for comparison.

Reporting in Taurus with Blazemeter

Blazemeter’s reporting feature is way better than any other we’ve seen so far. But running tests in their platform is a bit expensive. One solution is to use the Blazemeter reporting option in Taurus: we get all the advantages of their reporting capability without the cost of running tests in their servers.

However, one of its limitations is that the visualization period only lasts a week. In spite of that, it’s worth trying it in your projects. We have two alternatives: to execute the test from the console with the parameter “-report” (by which we get something looking like this: “bzt test.yml –report”), or to edit the configuration file and add the module “blazemeter” in the configuration section of the reporting.

It’s not necessary to log into Blazemeter to obtain reports, but it’s better to do so anyway in order to see all the reports in a single dashboard and compare directly in Blazemeter’s UI.

It’s also worth mentioning that if you run Taurus locally, a new tab opens up in the browser, where we can see the live execution of the test from Blazemeter’s UI. This functionality is very useful for professionals with a less technical profile. Personally, I prefer the live reporting offered in the console.

Test Summary by Blazemeter

There are plenty of options for visualizing the data provided by the tests, and some of them are very powerful. One of the simplest and most important is the one that generates the executive summary. Since the result is stored just for a week, I took a screenshot to show the data displayed.

As we can see, there are two graphics showing useful information:

  • Active users (or threads) and requests per second
  • Active users, response time and latency

Here we can see how, when the test was halfway through, response times started to look bad. We can ask ourselves why this happened: Why did the response time shoot up? Why are there errors in endpoints that weren’t supposed to fail? But it’s better to leave that for some other time.

This set of tools allows to create various types of tests. It all depends on what we are trying to cover. For example, we can create stress tests using the distributed testing functionality, or add assertions to the tests in order to check the content of the responses.

In Sum

Is this the best stack available? Probably not, but it was the best to meet our needs. We didn’t know other options either, except perhaps flood io.

Given that we needed to test performance, we considered JMeter the most stable solution. Despite being a bit unpolished, and the documentation somewhat confusing, it’s very robust.

We also needed to create reports that looked “nice” and that people who didn’t have a technical background could understand, so we considered using Blazemeter. While it was possible to run tests in their own infrastructure, we preferred to run tests in our own and use Blazemeter’s reporting feature (which they offer completely for free).

Lastly, we needed to reconfigure tests several times to get the most out of the infrastructure. Although that’s possible with JMeter, with Taurus you can do that more elegantly. At the same time, we could use it as a proxy to generate reports, so there wasn’t much to think about in that regard.

Even tough at first it seems that configuring the tools and obtain something that proves functional is complex, you can do things really quickly. As I was writing this article, we faced the need to measure the response times of a website containing 15 sections, and we got the reports in just 3 hours. That is why I recommend this stack for almost any project size you need to test.

Thanks for reading this and have a happy testing!

Fernando Lescano

He is one of our Tech Leaders and has been a developer at intive – FDV since March 2015. In the company, he works with technologies like Javascript ( NodeJS, AngularJS, MongoDB, Jquery, etc.), Python (Django), Java and PHP. He also takes part in the Frontend Team. He defines himself as “passionate about software development, but with many other interests”. Since December 2015 he has managed the development and maintenance of Java and of new JavaScript components (AngularJS). He is received a degree as a University Technician with a focus on software development and information technology from the Universidad Provincial de Ezeiza.

Add comment