Docker

Load Testing with JMeter Docker and AWS EC2

If you have an application or e-commerce site (or any site for that matter), it’s not uncommon that you would expect a higher level of traffic on certain days, like Black Friday.
At moments like these, we need to take our load tests to the next level and simulate larger numbers of concurrent users. If we are running our load tests locally with Apache JMeter™,
there are certain limitations to the number of users you can run, even if your computer has enough CPU and memory.To generate the heavy load by jmeter we run it in distributed mode.

Apache JMeter, an open source testing tool, is used for load testing, performance testing, and functional testing.

Challenges with Distributed Testing

When we talk about distributing JMeter, we refer to a Master-Slave architecture where JMeter uses Java RMI [Remote Method Invocation] to interact with objects in a distributed network. You can see this in the image below.

For a detailed description of a JMeter distributed testing setup and its usage, read the blog post  here. But the setup for distributed testing is a complex ,time consuming and error-prone procedure, based on the properties file and the command line. So why not to drastically reduce the setup complexity, by using Docker?

Why docker and AWS EC2?

  • To reduce setup complexity for distributed testing in JMeter.
  • To remove error-prone procedure based on the properties file and the command line.
  • To reduce the cost of infrastructure by using AWS EC2 instances for Master and Slaves machine which is pay-per-use kind of model which you can scale whenever you want.
  • Location independence ,testing from different geographical location.
  • Stimulating load tests containing numerous concurrent users from different geographies. 

Before we start the hands on through this tutorial ,to understand distributed testing using docker refer this. We are using same approach here but with AWS EC2 instance.

Infrastructure of Setup

We are going to create a distributed load testing infrastructure using AWS EC2, Docker and JMeter. Infrastructure will look like below:

The whole setup is divided into 2 parts ,Docker Images setup and AWS setup.

First we will create our own custom JMeter docker files as per our requirement then create images from the docker files and push them on docker hub to reuse them anytime anywhere.

Second, we will setup and configure AWS EC2 instances for our master and slave machines.

Prerequisites

  1. Some basic understanding of JMeter , Docker and AWS EC.
  2. A system having java and docker installed.
  3. An active AWS account.

Part-1 : Docker Images Setup

Step 1 : Create an Images

Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Docker can build images automatically by reading the instructions from a Dockerfile. The docker build command builds an image from a Dockerfile.

There are 3 images we need to create:

  1. JMeter Base Image : which creates the elementary setup required to run a JMeter instance.
  2. JMeter Master Image : On the top of base image ,JMeter Master image.
  3. JMeter Slave Image : On the top of base image ,JMeter Slave image .

Command to create a plain docker image:

docker build /path/to/dockerfile

Create a tag for a docker image:

docker tag imageId username/reponame:imageTag

Create an image and tag at the same time:

docker build -t username/reponame:imageTag /path/to/dockerfile

Dockerfile:

# Use Java 11 JDK Oracle Linux
FROM openjdk:11-jdk-oracle
MAINTAINER mnazim1541
# Set the JMeter version you want to use
ARG JMETER_VERSION="5.1.1"
# Set JMeter related environment variables
ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV JMETER_DOWNLOAD_URL https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-${JMETER_VERSION}.tgz
# Set default values for allocation of system resources (memory) which will be used by JMeter
ENV Xms 256m
ENV Xmx 512m
ENV MaxMetaspaceSize 1024m
# Change timezone to local time
ENV TZ="Europe/Bucharest"
RUN export TZ=$TZ
# Install jmeter
RUN yum -y install curl \
&& mkdir -p /tmp/dependencies \
&& curl -L --silent ${JMETER_DOWNLOAD_URL} >  /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz \
&& mkdir -p /opt \
&& tar -xzf /tmp/dependencies/apache-jmeter-${JMETER_VERSION}.tgz -C /opt \
&& rm -rf /tmp/dependencies
# Set JMeter home
ENV PATH $PATH:$JMETER_BIN
# copy our entrypoint
COPY entrypoint.sh /
RUN chmod +x ./entrypoint.sh
# Run command to allocate the default system resources to JMeter at 'docker run'
ENTRYPOINT ["/entrypoint.sh"]

Entrypoint.sh:

#!/bin/bash
# Run command to allocate the default or custom system resources (memory) to JMeter at 'docker run'
sed -i 's/\("${HEAP:="\)\(.*\)\("}"\)/\1-Xms'${Xms}' -Xmx'${Xmx}' -XX:MaxMetaspaceSize='${MaxMetaspaceSize}'\3/' ${JMETER_BIN}/jmeter
exec "$@"

On top of the base layer, you can create a Master layer and a Slave layer. These Dockerfiles can be customized according to your specific requirements.

Dockerfile for the Master layer:

# Use my custom base image defined above
FROM mnazim1541/jmeterrepo:jmeterbase
MAINTAINER Dragos
# Expose port for JMeter Master
EXPOSE 60000

Dockerfile for the Slave layer:

# Use my custom base image defined above
FROM mnazim1541/jmeterrepo:jmeterbase
MAINTAINER Dragos
# Expose ports for JMeter Slave
EXPOSE 1099 50000
COPY entrypoint.sh /
RUN chmod +x ./entrypoint.sh
# Run command to allocate the default system resources to JMeter at 'docker run' and start jmeter-server with all required parameters
ENTRYPOINT ["/entrypoint.sh"]

Entrypoint.sh for the Slave layer:

#!/bin/bash

# Run command to allocate the default system resources to JMeter at 'docker run'
sed -i 's/\("${HEAP:="\)\(.*\)\("}"\)/\1-Xms'${Xms}' -Xmx'${Xmx}' -XX:MaxMetaspaceSize='${MaxMetaspaceSize}'\3/' ${JMETER_BIN}/jmeter &&

$JMETER_HOME/bin/jmeter-server \
	-Dserver.rmi.localport=50000 \
	-Dserver_port=1099 \
	-Dserver.rmi.ssl.disable=true \
	-Djava.rmi.server.hostname=$HostIP

exec "$@"

The role of docker entrypoints is to initialise/ configure data into a container at runtime. In our case, we need them to specify how much memory JMeter is allowed to use and also to start the JMeter server preset with some custom configs required for our infrastructure to work.

Docker Commands to create Images:

Now let’s go through the command needed to create Docker images. As a side note, a Docker image represents a set of layers which ideally integrate well together and are a stable snapshot of the environment we need.

Command to create a plain docker image

docker build /path/to/dockerfile

Create a tag for a docker image:

docker tag imageId username/reponame:imageTag

Create an image and tag at the same time:

docker build -t username/reponame:imageTag /path/to/dockerfile

Step 2 : Create Containers

Now we have images ready , we can start creating containers which will provide our desired environment to run the performance scripts.

To create a new container:

docker run -dit --name containerName repository:tag or imageId /bin/bash

To start/stop a container:

docker start/stop containerId

To access a running container:

docker exec -it containerId or containerName /bin/bash

To watch the available container:

docker ps -a

Note: Check all the stuff on your local machine by running checking the jmeter version and running the jmeter script by command .

Step 3: Push/Pull Images to Dockerhub or any Private Docker Repo

Once you’ve tested that the created image meets the required criteria (everything worked inside the container), it’s generally a good idea to save this image to a repository. Then, you can just pull it from there whenever you need it without having to build it from a Dockerfile every time.

Push image to dockerhub:

docker push username/reponame:imageTag

Pull an existing image from dockerhub:

docker pull image_name : version

Now we are all set & ready to do setup on AWS cloud.

Part 2 : AWS EC2 Setup

Now its time to start the setup of a AWS EC2 instance to run of scripts , we are using AWS EC2 instance because it is scalable,easy to manage , pay per use model which saves lots of time ,money and effort in spinning up a machine.

You can use the EC2 free tier instances for up to 750 hours/ month for 1 year so that’s a lot of time to experiment.

Step 4 : Spinning /Setup of AWS EC2 Instance

Navigate to https://aws.amazon.com/ec2/ and create a free account.

Login into account and setup your region :

Go to All Services -> Compute -> EC2

Choose a Amazon machine image (AMI) – an OS which you want to install on your instance.

Select an instance type (t2.micro which is free ) and click to “Next Configure Instance Details” button.

Here we need to create and attache a IAM Policy .

Creating An IAM Policy:

 Lets assume you only require an infrastructure comprised of 1 JMeter master node and 2 slaves. In this case, it’s relatively easy to just access each instance and configure it (install docker + spin up the containers).

But what happens when you have more than 3 instances you have to work with?

Manually configuring each of them becomes tedious and probably not a good idea anymore. You will need a system which can manage the large amount of containers you are working with. Some well-known tools for this are Google’s Kubernetes or the tool called Rancher.

But since we are using AWS, these 2 solutions seem like over-engineering since Amazon offers an out of the box solution for specifically what we need to do.
The ‘Run Command’ feature allows us to execute shell scripts on multiple EC2 instances at the same time. So we don’t have to access each instance and install docker and start containers one instance at a time.

The only requirement for you to be able to execute a command on an EC2 instance via the ‘Run command’ feature is that the appropriate IAM role is associated to the instance. I named my IAM policy ‘EC2Command’ and I’m careful to select it for each newly created instance (but the role can be assigned to the instance later on just as well via the ‘attach/replace role’ function)

Setting an IAM policy to existing instance
Associating an IAM policy at instance creation

When you create the role, be sure to attach the ‘AmazonEC2RoleforSSM’ policy to your role and you should be good to go.

Associating a permission to the IAM role
Associating a permission to the IAM role

And that’s it, now you can use the ‘Run command’ feature to execute scripts on multiple instances bulk.

Attach IAM policy and Click “Add Storage”

Click “Next Add Tags”

Click “Next : Configure Security Group”

To enable JMeter instances (master or slave) inside the containers to communicate, a custom security group has been defined and attached to each host:

Create a Security Group

Inbound rules:

Security group inbound rules
Security group inbound rules

Outbound rules:

Security group outbound rules
Security group outbound rules

Note: be sure to assign all instances which you want to be part of the load testing infrastructure, to this security group or else they might not be able to communicate with each other.

Assign a security group and Click “Review And Launch” button

Click “Launch

From drop down select “Create a new key pair” and give a name to key pair.

Click “Launch Instances” and Hurray !! you instance is launching now !

Click on the instance ID and you will be navigated to Instances dashboard . As of now you can see the status of instance in pending as it is processing .

After some time it will be green(running status)

Step 5 : Install Docker on the Test Machines

You now need to install docker on your EC2 hosts so you can start spinning containers and connecting them together for the distributed load test.

Direct command (executed directly in an instance’s terminal ):

sudo apt-get install  curl  apt-transport-https ca-certificates software-properties-common \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add \
&& sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
&& sudo apt-get update \
&& sudo apt-get install -y docker-ce \
&& sudo usermod -aG docker $USER \
&& sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose \
&& sudo chmod +x /usr/local/bin/docker-compose \
&& sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Shell script to execute via ‘Run command’:

#!/bin/bash
sudo apt-get install  curl  apt-transport-https ca-certificates software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce
USER_DOCKER=$(getent passwd {1000..60000} | grep "/bin/bash" | awk -F: '{ print $1}')
sudo usermod -aG docker $USER_DOCKER

you would run this second script on multiple EC2 instances after which they will all have a working version of Docker. The next step is to configure the master and slave nodes.

Step 6 : Configure the Master Node

In some situations, you might not even need multiple slave nodes to run your tests distributedly. This applies when you have a powerful host machine and that machine is enough to generate the target load. So for this specific case, steps 7 and 8 are not required. For this situation, you might not even want to use a container and install JMeter directly on the host.

But let’s assume we need a Master + Slaves system and proceed to launching the Master container:

Direct command (executed directly in an instance’s terminal on Ubuntu):

HostIP=$(ip route show | awk '/default/ {print $9}') \
&& docker pull mnazim1541/jmeterrepo:jmetermaster \
&& docker run -dit --name master --network host -e HostIP=$HostIP -e Xms=256m -e Xmx=512m -e MaxMetaspaceSize=512m -v /opt/Sharedvolume:/opt/Sharedvolume mnazim1541/jmeterrepo:jmetermaster /bin/bash

Shell script to execute via ‘Run command’:

#!/bin/bash
HostIP=$(ip route show | awk '/default/ {print $9}')
docker pull mnazim1541/jmeterrepo:jmetermaster
docker run -dit --name master --network host -e HostIP=$HostIP -e Xms=256m -e Xmx=512m -e MaxMetaspaceSize=512m -v /opt/Sharedvolume:/opt/Sharedvolume mnazim1541/jmeterrepo:jmetermaster /bin/bash

The first line of the script stores the machines private IP in the variable ‘HostIP’. The master’s HostIP is not used for anything, only the HostIPs for the slave nodes will be used. We will see exactly what for at step 8.

The next line is straightforward and just fetches the image from the appropriate repo.

The last line creates the container we are going to use. There are a few points worth going over in this command:

  1. The ‘ — network host’ command enables host networking which means that applications inside the containers (JMeter), will be available on the ports which were exposed in the ‘entrypoint.sh’ script. ( The issue was that even though the script was being executed on the slave nodes, no results were aggregated on the master node because of an error (‘java . rmi . ConnectException: Connection refused to host:masterPrivateIP’). Note that I did not encounter this issue with older versions of JMeter like 3 . x . x.
  2. -e Xms=256m -e Xmx=512m -e MaxMetaspaceSize=512m’ is basically parameterization of the XmsXmx and MaxMetaspaceSize which dictates the amount of memory JMeter is allowed to use. This is done by first setting some environment variables inside the container. Then, running the command inside the ‘entrypoint.sh’ script which changes the ‘jmeter’ file found inside JMeter’s ‘/bin’ folder. If these values are not specified, a default is used.
  3. The ‘-v /opt/Sharedvolume:/opt/Sharedvolume userName/repoName: imageTag’ command just maps a folder on the host to a folder inside the container where you will have the script files and generated logs.

Step 7: Configure Slave Nodes

The ‘HostIP’ variable is only used here in the ‘entrypoint.sh’ script to enable remote access from the master to the slave (‘-Djava . rmi . server . hostname=$HostIP’).

Direct command (executed directly in an instance’s terminal on Ubuntu):

HostIP=$(ip route show | awk '/default/ {print $9}') \
&& docker pull mnazim1541/jmeterrepo:jmeterslave \
&& docker run -dit --name slave --network host -e HostIP=$HostIP -e HostIP=$HostIP -e Xms=256m -e Xmx=512m -e MaxMetaspaceSize=512m mnazim1541/jmeterrepo:jmeterslave /bin/bash

Shell script to execute via ‘Run command’:

#!/bin/bash
HostIP=$(ip route show | awk '/default/ {print $9}')
docker pull mnazim1541/jmeterrepo:jmeterslave
docker run -dit --name slave --network host -e HostIP=$HostIP -e Xms=256m -e Xmx=512m -e MaxMetaspaceSize=512m mnazim1541/jmeterrepo:jmeterslave /bin/bash

Step 8 : Run Script in Distributed Mode

Now we should have everything ready to start running our tests. Here are the command which we need to run on master node in order to start running the distributed tests:

jmeter -n -t /path/to/scriptFile.jmx -Dserver.rmi.ssl.disable=true -R host1PrivateIP, host2PrivateIP,..., hostNPrivateIP -l /path/to/logfile.jtl

References :

https://docs.aws.amazon.com/efs/latest/ug/gs-step-one-create-ec2-resources.html

https://docs.docker.com/

https://www.vinsguru.com/jmeter-distributed-load-testing-using-docker/

Categories: Docker, JMeter

Tagged as:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s