| Table of Contents | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Preface
ClinSpark is designed to be performant with the goal of…
...
We expect to see average response times in properly sized production environments around 250 ms.
Introduction
| Info |
|---|
The testing exercise reported here was conducted in 2016. |
We don’t typically engage in performance testing exercises with customers. If this is something you need, please talk to your project manager and they can arrange to have such an exercise scoped and costed.
...
| Info |
|---|
Raw data outputs from JMeter are available for the experiment described here but and have not been included as downloadable ZIP files within each test run results section. Download the ZIP, unpack it and click on the index.html file  |
Executive Summary
This exercise objectively demonstrates that ClinSpark maintains a very low response time even with large volumes of accumulated study data in the database.
...
The script was run against differing database sizes as described in the Results section that follows. For each test run, very detailed reports are generated and are available for download. High-level summaries of each test run are found with their corresponding tests in the sections that follow. APDEX or Application Performance Index (http://www.apdex.org) is a measure of user experience based on user wait times. The configured APDEX satisfaction and tolerance thresholds have been defined as 3 and 5 seconds respective.
...
Five (5) Studies
APDEX Score: 0.996
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 10,001 |
volunteer_medical_condition | 70,007 |
base_test_result | 144,576 |
item_data | 288,650 |
item_data_sample_audit_record | 346,380 |
item_data_audit_record | 578,053 |
Sum database table rows | 2,800,162 |
Reports
...
50 Studies
APDEX Score: 0.996
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 50,001 |
volunteer_medical_condition | 350,007 |
base_test_result | 1,440,576 |
item_data | 2,876,150 |
item_data_sample_audit_record | 3,451,380 |
item_data_audit_record | 5,759,803 |
Sum database table rows | 26,945,065 |
Reports
100 Studies
APDEX Score: 0.995
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 75,001 |
volunteer_medical_condition | 525,007 |
base_test_result | 2,880,576 |
item_data | 5,751,150 |
item_data_sample_audit_record | 6,901,380 |
item_data_audit_record | 11,517,303 |
Sum database table rows | 53,442,557 |
Reports
...
250 Studies
APDEX Score: 0.997
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 125,001 |
volunteer_medical_condition | 875,007 |
base_test_result | 7,200,576 |
item_data | 14,376,150 |
item_data_sample_audit_record | 17,251,380 |
item_data_audit_record | 28,789,803 |
Sum database table rows | 132,509,822 |
Reports
...
500 Studies
APDEX Score: 0.996
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 200,001 |
volunteer_medical_condition | 1,400,007 |
base_test_result | 14,400,576 |
item_data | 28,751,150 |
item_data_sample_audit_record | 34,501,380 |
item_data_audit_record | 57,577,303 |
Sum database table rows | 264,147,372 |
Reports
...
1000 Studies
APDEX Score: 0.994
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 350,000 |
volunteer_medical_condition | 2,449,995 |
base_test_result | 28,800,576 |
item_data | 57,501,150 |
item_data_sample_audit_record | 69,001,380 |
item_data_audit_record | 115,152,303 |
Sum database table rows | 527,422,350 |
Reports
...
1500 Studies
APDEX Score: 0.989
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 400,000 |
volunteer_medical_condition | 2,799,995 |
base_test_result | 43,200,576 |
item_data | 86,251,150 |
item_data_sample_audit_record | 103,501,380 |
item_data_audit_record | 172,727,304 |
Sum database table rows | 788,997,461 |
Reports
...
2000 Studies
APDEX Score: 0.985
| View file | ||
|---|---|---|
|
Key Data
Table | Row Count |
volunteer | 500,000 |
volunteer_medical_condition | 3,499,995 |
base_test_result | 57,600,576 |
item_data | 115,001,150 |
item_data_sample_audit_record | 138,001,380 |
item_data_audit_record | 230,302,303 |
Sum database table rows | 1,051,422,354 |
Reports
...
Test Components
Web Server
For the purpose of this test, only a single node was configured to host the ClinSpark application. Typical ClinSpark deployments have a minimum of two nodes and can scale infinitely horizontally as required.
Hosting provider: Amazon Web Services, region: US East
Server: instance type: m4.large, 2 vCPU, 8 GB RAM
Operating System: Linux 4.4.14-24.50.amzn1.x86_64 #1 SMP Fri Jun 24 19:56:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Server type: Apache Tomcat/7.0.69
Java: 1.7.0_101, OpenJDK Runtime Environment (amzn-2.6.6.1.67.amzn1-x86_64 u101-b00), OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
JVM Settings: -Xms1024M -Xmx1024M -XX:MaxPermSize=512M -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
Database Server
Hosting provider: Amazon Web Services, region: US East
Server: instance type: db.r3.2xlarge, 8 vCPU, 61 GB RAM
Test Harness Client
In order to have more realistic latency times, the test harness was executed on a service offering outside of the web server's network and infrastructure. Details are as follows:
Hosting provider: Digital Ocean
Operating System: Linux 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Java: 1.7.0_65, OpenJDK Runtime Environment (IcedTea 2.5.2) (7u65-2.5.2-3~14.04), OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
JMeter Script
Version: 3.0 r1743807
Analysis table definitions
volunteer - volunteers in the system that are not necessarily enrolled in any given study.
volunteer_medical_condition - a join table that combines volunteer records and dynamically associated medical conditions.
base_test_result - lab results
item_data - a data point associated with a form / item group. Examples are QT interval, PK capture time, etc.
item_data_sample_audit_record - during the process of study samples, audit records as established when users perform each step.
item_data_audit_record - audit records for item data are generated at creation time and each time a user interacts with a form in which the item exists.
Sum database table rows - this is a sum of all of the ClinSpark table rows. It is not the sum of the rows presented in the analysis section.
Miscellaneous
Linux operating system information obtained via shell command: uname -a
Tomcat information obtained via shell command: java -cp $TOMCAT_HOME/lib/catalina.jar org.apache.catalina.util.ServerInfo
Java version obtained via shell command: java -version
JMeter version obtained via shell command: $JMETER_HOME/bin/jmeter --version
Before each test was run, a 'warm up' of the database was initiated. This was done by way of executing the test scripts and allowing a user to click through the application. Approximate warm up time was ten minutes of test execution.
Each test assumes that no more than 50 studies are in an 'active' state.
Postscript
This testing exercise was executed when ClinSpark was being pitched in 2016, to a prospective customer that was also evaluating another eSource platform. We were aware that this competitor eSource platform, from an established multinational vendor, was poorly performant and we were happy to go head-to-head to demonstrate ClinSpark’s designed-in performance characteristics.
...