Why performance test Jira?
If you are in charge of an enterprise instance of Jira, it may pay off to understand its performance characteristics well and performance test Jira frequently, in particular for planning your growth and before you add in additional data for example when you try to merge instances. We are an App vendor on the Atlassian Marketplace, well-known for our Mobility for Jira iOS and Android mobile apps and Google Sheets Integration app. We have many large clients and our largest client for example has more than 100k licensed users. So we are very experienced working with large Jira instances and optimizing add-ons and apps constantly for performance. We want to briefly share our experience setting up and testing a large Jira Data Center instance on Amazon’s AWS. This also applies to an on-prem system or running on Microsoft’s Azure platform.
How to set up a Jira Data Center instance on AWS
We used the Atlassian Quickstart Template for Jira Data Center. This template is based on CloudFormation and provides some configuration options to define how your instance will be set up. For example you can configure the size and number of application EC2 nodes and the size of the database nodes. For larger deployments eventually you want the nodes to auto-scale. You can install the Jira Data Center stack into an existing VPC or a new VPC. To make sure there wasn’t any unexpected interference, we chose to install Jira into a new VPC. For the initial set up we kept the number of nodes to one but decided to choose larger nodes than just the minimum size since we encountered some initial problems with unresponsiveness. We chose a c3.xlarge Jira node instance type and a m4.2xlarge database instance type. Also we increased the storage size to 100GB. Here are some suggestions by Atlassian on EC2 node types. After finishing the configuration we let CloudFormation do its magic and create the stack.
CloudFormation provides an output with the load balancer URL. This is the URL to navigate to via browser after the setup. You can always find it in the AWS CloudFormation Console. To complete the setup, you will also need a Jira Data Center evaluation license from the Atlassian Portal. You can navigate to the load balancer URL and set up Jira Data Center as you would a normal Jira instance.
Now that the instance is available we needed to populate it with data since there is no point in performance testing an instance with no data. You may choose to restore a backup from your large production instance or you can, as we did, create simulated data. For this purpose we used the Jira Data Generator app, freely available from the Atlassian Marketplace. Depending on your requirements you may choose different parameters but we generated the following data: 2000 projects, 100,000 users, 50,000 groups, 1,600 custom fields, and 2,000,000 issues. This is what Atlassian considers a large Jira instance. Never use the Data Generator in production instance. It may corrupt your set up. We encountered some issues during data generation, in which we received some error messages. After restarting the EC2 node, let garbage collection complete, and re-indexing of the system all the data seemed to be there.
Creating and Running Performance Tests
Now that you have large instance readily available it is time to create some tests. We used Apache’s JMeter to define some performance test cases and then run them and step up the number of concurrent users. There are variety of other tools that can be used for that purpose. The tests reflect mostly the RESTful API calls that are made by our apps from iOS and Android mobile devices. Test runs over the network my deviate from each other due to many other factors playing a role such as network latency. It pays off to take at least 10-50 samples for test case and use the Median instead of Average to remove outliers or use the 95th percentile.
We encountered several issues during these tests. Mostly related to the Jira Data Center set up. We hoped the CloudFormation template provided by Atlassian was more mature. One problem that we encountered was that it isn’t easy to increase the heap size. Our instance was running out of heap memory when scaling up/down. We had to increase the heap memory to 4096M for each node manually. Unfortunately a defect doesn’t allow you to make the change in the template itself. Definitely a blocker for using the template for large production data center instances.
Performance testing with Jira Data Center for large enterprise installations is essential. AWS may provide a quick way of spinning up an environment and test its performance characteristics. If you are running Jira own your own hardware, the results may not be comparable and you may need to create an environment in your own data center. AWS also provides some additional testing capabilities such as AWS Mobile Device Farm for testing. Atlassian also provides another way for performance testing via JPT, which is a Maven-based framework to spin up an environment and run tests. This framework is something to be watched but we felt it wasn’t ready for prime time yet.