In this part we will talk about test results and charts which are available out of the box.
But first I would like to notice a change in the toolkit which really makes test monitoring much easier. Now while the test is running you can see metrics right in your terminal window:
It is called Taurus console reporter. This screen is divided into three parts: test information represented by graphs, test information in text and your system information.
System information is a very important section. Because if during a test run you use 100% cpu, it can mean that the load you are trying to put on your Jira, Confluence or Bitbucket instance is not as much as you intended to put. Because your machine does not have enough resources to generate the intended load.
Test Results
I will talk about test results based on a test run for a Confluence instance.
After your test has been completed, you will have the following files in the app/results/confluence:
Let’s talk about each file in detail.
test_1_selenium_create_comment-2020-07-24-14-37-36.853895.html
error_artifacts folder contains information about errors in Selenim tests. For example, here is the test_1_selenium_create_comment-2020-07-24-14-37-36.853895.html:
<html><head>
<title>Selenium - UobuX2FuIO - TES - Product Team Confluence</title>
<meta http-equiv="X-UA-Compatible" content="IE=EDGE,chrome=IE7">
<meta charset="UTF-8">
<meta id="confluence-context-path" name="confluence-context-path" content="/confluence">
<meta id="confluence-base-url" name="confluence-base-url" content="https://my.domain.com/confluence">
<meta id="atlassian-token" name="atlassian-token" content="f83db95dc7c654e23f789178b7ed6b7e98d1077b">
<meta id="confluence-cluster-node-id" name="confluence-cluster-node-id" value="37863810">
<meta id="confluence-cluster-node-name" name="confluence-cluster-node-name" value="node3">
<meta id="confluence-space-key" name="confluence-space-key" content="TES">
<script type="text/javascript">
var contextPath = '/confluence';
</script>
<meta name="robots" content="noindex,nofollow">
<meta name="robots" content="noarchive">
<meta name="confluence-request-time" content="1595590631816">
..........As you can see it is the html code of the page, which caused the error. In the test_1_selenium_create_comment-2020-07-24-14-37-36.853895.png file we can see a screenshot of the page:
We have “Unable to communicate with server” error. This information is very useful for understanding why the error happened. In reality we had the java.lang.OutOfMemoryError: Java heap space and the server went down.
bzt.log
This file contains information which you can see in the terminal while running tests:
[2020-07-24 14:32:11,645 INFO root] Taurus CLI Tool v1.14.2
[2020-07-24 14:32:11,646 DEBUG root] Command-line options: {'log': '/var/folders/4r/w8hkd4w51x787tw4ynb29v2w0000gn/T/bzt_deodyr86.log', 'option': None, 'quiet': None, 'verbose': None, 'no_system_configs': None, 'aliases': []}
[2020-07-24 14:32:11,646 DEBUG root] Python: CPython 3.8.5
[2020-07-24 14:32:11,646 DEBUG root] OS: uname_result(system='Darwin', node='Alexeys-MacBook-Pro-2.local', release='19.6.0', version='Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64', machine='x86_64', processor='i386')
[2020-07-24 14:32:11,646 INFO root] Starting with configs: ['confluence.yml']
[2020-07-24 14:32:11,649 DEBUG root] Using personal config: /Users/alexm/.bzt-rc
[2020-07-24 14:32:11,649 INFO Engine] Configuring...
[2020-07-24 14:32:11,718 DEBUG Engine] Base configs list: ['/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/venv/lib/python3.8/site-packages/bzt/resources/10-base-config.yml']
[2020-07-24 14:32:11,719 DEBUG Engine.Configuration] Configs: ['/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/venv/lib/python3.8/site-packages/bzt/resources/10-base-config.yml']
[2020-07-24 14:32:11,719 DEBUG Engine.Configuration] Reading /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/venv/lib/python3.8/site-packages/bzt/resources/10-base-config.yml as YAML
[2020-07-24 14:32:11,742 DEBUG Engine] User configs list: ['/Users/alexm/.bzt-rc', 'confluence.yml']
[2020-07-24 14:32:11,742 DEBUG Engine.Configuration] Configs: ['/Users/alexm/.bzt-rc', 'confluence.yml']
[2020-07-24 14:32:11,743 DEBUG Engine.Configuration] Reading /Users/alexm/.bzt-rc as YAML
[2020-07-24 14:32:11,747 DEBUG Engine.Configuration] Reading confluence.yml as YAML
[2020-07-24 14:32:11,762 DEBUG Engine.Configuration] Configs: ['/Users/alexm/.bzt-rc', 'confluence.yml']
[2020-07-24 14:32:11,762 DEBUG Engine.Configuration] Reading /Users/alexm/.bzt-rc as YAML
[2020-07-24 14:32:11,766 DEBUG Engine.Configuration] Reading confluence.yml as YAML
[2020-07-24 14:32:11,781 DEBUG Engine] Requesting updates info: https://gettaurus.org/updates/?version=1.14.2&installID=macos-acde48001122
[2020-07-24 14:32:11,781 DEBUG HTTPClient] Making HTTP request GET https://gettaurus.org/updates/?version=1.14.2&installID=macos-acde48001122
[2020-07-24 14:32:11,781 INFO Engine] Artifacts dir: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11
[2020-07-24 14:32:11,783 DEBUG Engine] New artifact filename: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/effective
[2020-07-24 14:32:11,785 DEBUG Engine.Configuration] Dumping YAML config into /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/effective.yml
[2020-07-24 14:32:11,807 DEBUG Engine.Configuration] Dumping JSON config into /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/effective.json
[2020-07-24 14:32:11,808 DEBUG Engine] New artifact filename: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/merged.yml
[2020-07-24 14:32:11,809 DEBUG Engine.Configuration] Dumping YAML config into /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/merged.yml
[2020-07-24 14:32:11,816 DEBUG Engine] New artifact filename: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/merged.json
[2020-07-24 14:32:11,817 DEBUG Engine.Configuration] Dumping JSON config into /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/merged.json
[2020-07-24 14:32:11,817 DEBUG Engine] Add existing artifact (move=False): confluence.yml
[2020-07-24 14:32:11,818 DEBUG Engine] Copying confluence.yml to /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/confluence.yml
[2020-07-24 14:32:11,819 DEBUG root] Closing log handler: /var/folders/4r/w8hkd4w51x787tw4ynb29v2w0000gn/T/bzt_deodyr86.log
[2020-07-24 14:32:11,819 DEBUG root] Switched writing logs to /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_14-32-11/bzt.log
[2020-07-24 14:32:11,819 DEBUG root] Linting config
[2020-07-24 14:32:11,823 INFO Engine] Preparing...
........................
If you need to see how the prepare, execute and post run stages went, you should have a look at this file.
confluence.yml
This file is a copy of confluence.yml which you used to run the test with bzt confluence.yml command. It is very convenient to have this file to check which parameters were used for the test.
effective.json/effective.yml
These two files are copies of confluence.yml file in yml and json formats
error.jtl
You can use this file for debugging Jmeter scripts. You can find information on how to do it in the docs/confluence/README.md file.
jmeter.err
Jmeter error log. In my case this file is empty.
jmeter.log
Jmeter log file which contains all log messages sent by Jmeter during test execution:
2020-07-24 15:10:00,962 INFO o.a.j.u.JMeterUtils: Setting Locale to en_RU
2020-07-24 15:10:00,978 INFO o.a.j.JMeter: Loading user properties from: /Users/alexm/.bzt/jmeter-taurus/5.2.1/bin/user.properties
2020-07-24 15:10:00,979 INFO o.a.j.JMeter: Loading system properties from: /Users/alexm/.bzt/jmeter-taurus/5.2.1/bin/system.properties
2020-07-24 15:10:00,979 INFO o.a.j.JMeter: Loading additional properties from: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_15-07-19/jmeter-bzt.properties
2020-07-24 15:10:00,979 INFO o.a.j.JMeter: Setting System properties from file: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_15-07-19/system.properties
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: Copyright (c) 1998-2019 The Apache Software Foundation
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: Version 5.2.1
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: java.version=1.8.0_252
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: java.vm.name=OpenJDK 64-Bit Server VM
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: os.name=Mac OS X
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: os.arch=x86_64
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: os.version=10.15.6
2020-07-24 15:10:00,980 INFO o.a.j.JMeter: file.encoding=UTF-8
2020-07-24 15:10:00,981 INFO o.a.j.JMeter: java.awt.headless=true
2020-07-24 15:10:00,981 INFO o.a.j.JMeter: Max memory =7635730432
2020-07-24 15:10:00,981 INFO o.a.j.JMeter: Available Processors =8
2020-07-24 15:10:00,984 INFO o.a.j.JMeter: Default Locale=English (Russia)
2020-07-24 15:10:00,984 INFO o.a.j.JMeter: JMeter Locale=English (Russia)
2020-07-24 15:10:00,984 INFO o.a.j.JMeter: JMeterHome=/Users/alexm/.bzt/jmeter-taurus/5.2.1
2020-07-24 15:10:00,984 INFO o.a.j.JMeter: user.dir =/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app
2020-07-24 15:10:00,984 INFO o.a.j.JMeter: PWD =/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app
2020-07-24 15:10:05,994 INFO o.a.j.JMeter: IP: 192.168.0.33 Name: Alexeys-MacBook-Pro-2.local FullName: 192.168.0.33
2020-07-24 15:10:05,995 INFO o.a.j.JMeter: user.classpath=/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_15-07-19:/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/jmeter
2020-07-24 15:10:05,995 INFO o.a.j.JMeter: Adding to classpath and loader: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_15-07-19
2020-07-24 15:10:05,996 INFO o.a.j.JMeter: Adding to classpath and loader: /Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/jmeter
2020-07-24 15:10:06,002 INFO o.a.j.s.FileServer: Default base='/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app'
2020-07-24 15:10:06,006 INFO o.a.j.s.FileServer: Set new base='/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/jmeter'
2020-07-24 15:10:06,118 INFO o.a.j.s.SaveService: Testplan (JMX) version: 2.2. Testlog (JTL) version: 2.2
2020-07-24 15:10:06,126 INFO o.a.j.s.SaveService: Using SaveService properties file encoding UTF-8
2020-07-24 15:10:06,130 INFO o.a.j.s.SaveService: Using SaveService properties version 5.0
................jmeter-bzt.properties
eter.save.saveservice.autoflush=true
jmeter.save.saveservice.connect_time=true
jmeterengine.force.system.exit=true
summariser.name=
application_hostname=my.domain.io
application_protocol=https
application_port=443
application_postfix=/confluence
total_actions_per_hr=200
perc_view_page=54
perc_view_dashboard=6
perc_view_blog=8
perc_search_cql=7
perc_create_blog=3
perc_create_and_edit_page=6
perc_comment_page=5
perc_view_attachment=3
perc_upload_attachment=5
perc_like_page=3
perc_standalone_extension=0
jmeterengine.nongui.port=4445
jmeterengine.nongui.maxport=4445
jmeter.save.saveservice.timestamp_format=ms
sampleresult.default.encoding=UTF-8
user.classpath=/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2020-07-24_15-07-19:/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/jmeter
This file contains all parameters with which Jmeter ran during test execution.
kpi.jtl
timeStamp,elapsed,label,responseCode,responseMessage,threadName,success,bytes,grpThreads,allThreads,Latency,Hostname,Connect
1595592651980,13982,jmeter_login_and_view_dashboard,200,"Number of samples in transaction : 8, number of failing samples : 0",Confluence 1-1,true,20910,1,1,0,Alexeys-MacBook-Pro-2.local,0
1595592666301,20109,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-1,true,34575,1,1,0,Alexeys-MacBook-Pro-2.local,0
1595592686489,983,jmeter_view_page_tree,404,,Confluence 1-1,false,32539,1,1,392,Alexeys-MacBook-Pro-2.local,0
1595592738274,16580,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-1,true,34564,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592754868,805,jmeter_view_page_tree,404,,Confluence 1-1,false,32539,2,2,411,Alexeys-MacBook-Pro-2.local,0
1595592742430,15346,jmeter_login_and_view_dashboard,200,"Number of samples in transaction : 8, number of failing samples : 0",Confluence 1-2,true,20958,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592774276,36189,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-1,true,34581,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592810477,30908,jmeter_view_page_tree,Non HTTP response code: java.net.SocketTimeoutException,Non HTTP response message: Read timed out,Confluence 1-1,false,2267,2,2,10712,Alexeys-MacBook-Pro-2.local,590
1595592829809,20611,jmeter_view_page,Non HTTP response code: java.net.SocketTimeoutException,,Confluence 1-2,false,2905,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592841398,20907,jmeter_recently_viewed,Non HTTP response code: java.net.SocketTimeoutException,"Number of samples in transaction : 1, number of failing samples : 1",Confluence 1-1,false,2905,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592862307,19672,jmeter_search_results,200,"Number of samples in transaction : 2, number of failing samples : 0",Confluence 1-1,true,2091,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592850434,32843,jmeter_login_and_view_dashboard,Non HTTP response code: java.net.SocketTimeoutException,"Number of samples in transaction : 8, number of failing samples : 3",Confluence 1-3,false,20394,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592883305,10416,jmeter_login_and_view_dashboard,200,"Number of samples in transaction : 8, number of failing samples : 0",Confluence 1-4,true,20932,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592882057,15549,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-1,true,34577,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592897621,699,jmeter_view_page_tree,404,,Confluence 1-1,false,32539,2,2,306,Alexeys-MacBook-Pro-2.local,0
1595592898329,3847,jmeter_view_dashboard,200,"Number of samples in transaction : 7, number of failing samples : 0",Confluence 1-1,true,19843,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592902229,16529,jmeter_create_page_editor,404,"Number of samples in transaction : 11, number of failing samples : 1",Confluence 1-1,false,58991,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592918772,8326,jmeter_create_page,200,"Number of samples in transaction : 17, number of failing samples : 0",Confluence 1-1,true,40246,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592927111,4573,jmeter_open_editor,404,"Number of samples in transaction : 9, number of failing samples : 1",Confluence 1-1,false,56905,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592929738,7926,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-4,true,34561,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592931695,6455,jmeter_edit_page,200,"Number of samples in transaction : 17, number of failing samples : 0",Confluence 1-1,true,40055,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592937682,745,jmeter_view_page_tree,404,,Confluence 1-4,false,32539,2,2,339,Alexeys-MacBook-Pro-2.local,0
1595592938438,942,jmeter_comment,200,"Number of samples in transaction : 1, number of failing samples : 0",Confluence 1-4,true,1316,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592939441,3717,jmeter_upload_attachment,200,"Number of samples in transaction : 1, number of failing samples : 0",Confluence 1-4,true,13991,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592965749,5643,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-4,true,34558,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592971405,674,jmeter_view_page_tree,404,,Confluence 1-4,false,32539,2,2,270,Alexeys-MacBook-Pro-2.local,0
1595592974241,7707,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-1,true,34558,2,2,0,Alexeys-MacBook-Pro-2.local,0
1595592981957,649,jmeter_view_page_tree,404,,Confluence 1-1,false,32539,2,2,252,Alexeys-MacBook-Pro-2.local,0
1595593010250,5336,jmeter_view_page,200,"Number of samples in transaction : 13, number of failing samples : 0",Confluence 1-1,true,34576,2,2,0,Alexeys-MacBook-Pro-2.local,0
...............................Jmeter raw data. This file contains information about requests and responses of JMeter during test run. Later this file will be used by this toolkit to create a summary statistic for the test run.
merged.yml/merged.json
Taurus can be run with multiple configuration files and the merged.json/merged.yml files contain merged data from all these configuration files. In our case merged.json/merged.yml will equal to effective.json/effective.yml.
modified_confluence.jmx
This file contains Jmeter project file with all settings and parameters applied. This file is executed for running Jmeter tests.
pytest.err/pytest.out
Raw files of the pytest engine. Pytest engine is used for Selenium tests.
results.csv
Summary of the results for the test run. It contains information about response time for all selenium and jmeter tests:
Label,# Samples,Average,Median,90% Line,95% Line,99% Line,Min,Max,Error %
jmeter_login_and_view_dashboard,4,15996,12216,29332,31667,33536,5548,34003,25.0
jmeter_view_page,18,9950,7290,21767,25701,31178,1041,32547,11.0
jmeter_view_page_tree,16,3422,1001,10405,13115,19515,700,21115,100.0
jmeter_recently_viewed,2,5068,5068,8533,8966,9312,738,9399,0.0
jmeter_search_results,2,21082,21082,36756,38716,40283,1488,40675,50.0
jmeter_view_blog,2,21786,21786,30568,31666,32544,10807,32764,50.0
jmeter_view_dashboard,2,6070,6070,6424,6468,6503,5629,6512,0.0
jmeter_create_page_editor,2,6514,6514,7286,7382,7460,5549,7479,100.0
jmeter_create_page,2,12844,12844,17641,18240,18720,6848,18840,0.0
jmeter_open_editor,2,20412,20412,32523,34037,35248,5272,35551,100.0
jmeter_edit_page,2,45840,45840,76536,80373,83443,7470,84210,50.0
jmeter_comment,2,1176,1176,1293,1307,1319,1029,1322,0.0
jmeter_upload_attachment,2,3639,3639,3922,3958,3986,3285,3993,0.0
jmeter_create_blog_editor,1,7213,7213,7213,7213,7213,7213,7213,100.0
jmeter_create_blog,1,15801,15801,15801,15801,15801,15801,15801,0.0
jmeter_view_attachment,1,1000,1000,1000,1000,1000,1000,1000,0.0
jmeter_like_page,1,2020,2020,2020,2020,2020,2020,2020,0.0
selenium_login:open_login_page,17,1355,538,2404,5970,9466,342,10340,0.0
selenium_login:login_and_view_dashboard,17,1977,1660,2917,3929,7006,1027,7775,0.0
selenium_login,17,3456,2317,5472,10064,16635,1520,18278,0.0
selenium_view_page,17,2310,734,3046,8832,19338,599,21964,0.0
selenium_create_page:open_create_page_editor,17,5715,1623,7635,23837,50605,1341,57297,0.0
selenium_create_page:save_created_page,17,5462,3789,10068,17413,19791,3056,20385,0.0
selenium_create_page,17,12186,6244,23697,33888,66310,5331,74415,0.0
selenium_edit_page:open_create_page_editor,16,4595,3460,8714,13476,14733,2,15047,0.0
selenium_edit_page:save_edited_page,15,3365,2567,4053,6707,10897,1975,11945,0.0
selenium_edit_page,16,8029,6406,15966,16456,17122,2,17288,0.0
selenium_create_comment:write_comment,15,2522,2147,4070,4833,4875,1813,4886,0.0
selenium_create_comment:save_comment,15,1251,1146,1652,1669,1685,1120,1689,0.0
selenium_create_comment,16,4187,3982,6027,6661,7047,2,7143,0.0
selenium_view_blog,16,7152,4360,13916,16767,23327,2,24967,0.0
selenium_view_dashboard,16,5155,3646,5286,12210,27374,1,31165,0.0
selenium_log_out,16,5115,862,1096,18352,59433,1,69703,0.0results_summary.log
This file contains a short log of the test run:
Summary run status FAIL
Artifacts dir 2020-07-24_14-32-11
OS macOS
DC Apps Performance Toolkit version 3.0.1
Application confluence 7.4.1
Dataset info 975425 pages
Application nodes count 2
Concurrency 2
Expected test run duration from yml file 2700 sec
Actual test run duration 793 sec
Finished (False, 'Actual test duration 793 sec < than expected test_duration 2700 sec.')
Compliant (False, 'Test run duration 793 sec < than minimum test duration 2700 sec. Test run concurrency 2 < than minimum test concurrency 200.')
Success (False, 'One or more actions have success rate < 95.0 %.')
Action Success Rate Status
jmeter_comment 100.0 OK
jmeter_create_blog 100.0 OK
jmeter_create_blog_editor 0.0 Fail
jmeter_create_page 100.0 OK
jmeter_create_page_editor 0.0 Fail
jmeter_edit_page 50.0 Fail
jmeter_like_page 100.0 OK
jmeter_login_and_view_dashboard 75.0 Fail
jmeter_open_editor 0.0 Fail
jmeter_recently_viewed 100.0 OK
jmeter_search_results 50.0 Fail
jmeter_upload_attachment 100.0 OK
jmeter_view_attachment 100.0 OK
jmeter_view_blog 50.0 Fail
jmeter_view_dashboard 100.0 OK
jmeter_view_page 88.89 Fail
jmeter_view_page_tree 0.0 Fail
selenium_a_login 100.0 OK
selenium_create_comment 93.75 Fail
selenium_create_page 94.12 Fail
selenium_edit_page 93.75 Fail
selenium_view_blog 93.75 Fail
selenium_view_dashboard 87.5 Fail
selenium_view_page 100.0 OK
selenium_z_log_out 93.75 Fail
selenium.err
This file contains information about all errors with Selenium webdriver:
Action: test_1_selenium_create_page, Error: webdriver = <selenium.webdriver.chrome.webdriver.WebDriver (session="a14471438bdc54e10792c212a59928c0")>
confluence_datasets = {'blog_id': '52215665', 'blogs': [['52377239', 'TES'], ['52215590', 'TES'], ['52215591', 'TES'], ['52540654', 'TES'], ...'], ['46780039', 'TES'], ['46777426', 'TES'], ['50383897', 'TES'], ['47037799', 'TES'], ['47037475', 'TES'], ...], ...}
confluence_screen_shots = None
def test_1_selenium_create_page(webdriver, confluence_datasets, confluence_screen_shots):
> modules.create_confluence_page(webdriver, confluence_datasets)
selenium_ui/confluence_ui.py:15:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
selenium_ui/confluence/modules.py:100: in create_confluence_page
measure()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
@functools.wraps(func)
def wrapper():
if LOGIN_ACTION_NAME in interaction:
globals.login_failed = False
if globals.login_failed:
pytest.skip(f"login is failed")
start = time.time()
error_msg = 'Success'
full_exception = ''
try:
func()
success = True
except Exception:
success = False
# https://docs.python.org/2/library/sys.html#sys.exc_info
exc_type, full_exception = sys.exc_info()[:2]
error_msg = f"Failed measure: {interaction} - {exc_type.__name__}"
end = time.time()
timing = str(int((end - start) * 1000))
with open(selenium_results_file, "a+") as jtl_file:
timestamp = round(time.time() * 1000)
jtl_file.write(f"{timestamp},{timing},{interaction},,{error_msg},,{success},0,0,0,0,,0\n")
print(f"{timestamp},{timing},{interaction},{error_msg},{success}")
if not success:
if LOGIN_ACTION_NAME in interaction:
globals.login_failed = True
> raise Exception(error_msg, full_exception)
E Exception: ('Failed measure: selenium_create_page - Exception', Exception('Failed measure: selenium_create_page:save_created_page - TimeoutException', TimeoutException("Error in wait_until: Timed out after 20 sec waiting for <selenium.webdriver.support.expected_conditions.visibility_of_element_located object at 0x10eb8f7f0>. \nLocator: ('id', 'title-text')", None, None)))
selenium_ui/conftest.py:118: Exceptionselenium.jtl
This file contains raw information about selenium tests in the test run:
timeStamp,elapsed,label,responseCode,responseMessage,threadName,success,bytes,grpThreads,allThreads,Latency,Hostname,Connect
1595590485432,4877,selenium_login:open_login_page,,Success,,True,0,0,0,0,,0
1595590488565,2967,selenium_login:login_and_view_dashboard,,Success,,True,0,0,0,0,,0
1595590488565,8011,selenium_login,,Success,,True,0,0,0,0,,0
1595590494133,5549,selenium_view_page,,Success,,True,0,0,0,0,,0
1595590509607,15472,selenium_create_page:open_create_page_editor,,Success,,True,0,0,0,0,,0
1595590517792,5666,selenium_create_page:save_created_page,,Success,,True,0,0,0,0,,0
1595590517793,23658,selenium_create_page,,Success,,True,0,0,0,0,,0After the test run you should have a look at the results.csv and results_summary.log files to understand if the test run was successful or not. If you can see errors in those files then you can investigate the cause of the errors by looking at the Jmeter and Selenium error files.
Reports
Besides the files above you can also generate reports. Those reports are aimed for approving your app for Data Center.
Performance Report
This report will show you how your Jira, Confluence or Bitbucket instance ran with and without your app. First you make a test run with your app installed and then you make a test run without your app. After that you provides links to both of the results in the performance_profile.yml file:
# Defines which column from test runs is used for aggregated report. Default is "90% Line"
column_name: "90% Line"
runs:
# fullPath should contain a full path to the directory with run results. E.g. /home/$USER/dc-app-performance-toolkit/jira/results/2019-08-06_17-41-08
- runName: "without app"
fullPath: "/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/noplugin"
- runName: "with app"
fullPath: "/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/4_nodes"
# Chart generation config
index_col: "Action"
title: "DCAPT Performance Testing"
image_height_px: 1000
image_width_px: 1200Then you run the following command:
python3 csv_chart_generator.py performance_profile.ymlThe report contains three files: performance_profile_summary.log, performance_profile.csv, performance_profile.png
performance_profile_summary.log contains summary information about the two test runs:
cenario status: OK
========================================================================================================================
Run name: without app
Summary run status OK
Artifacts dir 2020-07-24_13-29-26
OS macOS
DC Apps Performance Toolkit version 3.0.1
Application confluence 7.4.1
Dataset info 975407 pages
Application nodes count 4
Concurrency 200
Expected test run duration from yml file 2700 sec
Actual test run duration 2710 sec
Finished (True, 'OK')
Compliant (True, 'OK')
Success (True, 'OK')
Action Success Rate Status
jmeter_comment 100.0 OK
jmeter_create_blog 100.0 OK
jmeter_create_blog_editor 100.0 OK
jmeter_create_page 100.0 OK
jmeter_create_page_editor 100.0 OK
jmeter_edit_page 100.0 OK
jmeter_like_page 100.0 OK
jmeter_login_and_view_dashboard 100.0 OK
jmeter_open_editor 100.0 OK
jmeter_recently_viewed 100.0 OK
jmeter_search_results 100.0 OK
jmeter_upload_attachment 100.0 OK
jmeter_view_attachment 100.0 OK
jmeter_view_blog 100.0 OK
jmeter_view_dashboard 100.0 OK
jmeter_view_page 100.0 OK
jmeter_view_page_tree 100.0 OK
selenium_a_login 100.0 OK
selenium_create_comment 100.0 OK
selenium_create_page 100.0 OK
selenium_edit_page 100.0 OK
selenium_view_blog 100.0 OK
selenium_view_dashboard 100.0 OK
selenium_view_page 100.0 OK
selenium_z_log_out 100.0 OK
========================================================================================================================
Run name: with app
Summary run status OK
Artifacts dir 2020-07-24_13-29-26
OS macOS
DC Apps Performance Toolkit version 3.0.1
Application confluence 7.4.1
Dataset info 975407 pages
Application nodes count 4
Concurrency 200
Expected test run duration from yml file 2700 sec
Actual test run duration 2710 sec
Finished (True, 'OK')
Compliant (True, 'OK')
Success (True, 'OK')
Action Success Rate Status
jmeter_comment 100.0 OK
jmeter_create_blog 100.0 OK
jmeter_create_blog_editor 100.0 OK
jmeter_create_page 100.0 OK
jmeter_create_page_editor 100.0 OK
jmeter_edit_page 100.0 OK
jmeter_like_page 100.0 OK
jmeter_login_and_view_dashboard 100.0 OK
jmeter_open_editor 100.0 OK
jmeter_recently_viewed 100.0 OK
jmeter_search_results 100.0 OK
jmeter_upload_attachment 100.0 OK
jmeter_view_attachment 100.0 OK
jmeter_view_blog 100.0 OK
jmeter_view_dashboard 100.0 OK
jmeter_view_page 100.0 OK
jmeter_view_page_tree 100.0 OK
selenium_a_login 100.0 OK
selenium_create_comment 100.0 OK
selenium_create_page 100.0 OK
selenium_edit_page 100.0 OK
selenium_view_blog 100.0 OK
selenium_view_dashboard 100.0 OK
selenium_view_page 100.0 OK
selenium_z_log_out 100.0 OK
========================================================================================================================
performance_profile.csv contains summary information about response time for each of the Selenium and Jmeter tests:
Action,without app,with app
jmeter_login_and_view_dashboard,92764,90969
jmeter_view_page,20174,21212
jmeter_view_page_tree,1027,1182
jmeter_recently_viewed,6537,6309
jmeter_search_results,20103,20091
jmeter_view_blog,20552,19301
jmeter_comment,1497,1581
jmeter_upload_attachment,3758,3895
jmeter_create_blog_editor,6201,6335
jmeter_create_blog,12925,11319
jmeter_view_attachment,1394,1281
jmeter_like_page,798,866
jmeter_view_dashboard,4971,4775
jmeter_create_page_editor,8401,8508
jmeter_create_page,8620,8801
jmeter_open_editor,4513,4541
jmeter_edit_page,9528,9225
selenium_login:open_login_page,703,790
selenium_login:login_and_view_dashboard,2115,2057
selenium_login,3007,3132
selenium_view_page,1065,1269
selenium_create_page:open_create_page_editor,9562,9378
selenium_create_page:save_created_page,5603,5202
selenium_create_page,14663,14980
selenium_edit_page:open_create_page_editor,4059,3950
selenium_edit_page:save_edited_page,2965,2870
selenium_edit_page,6393,6959
selenium_create_comment:write_comment,4763,4575
selenium_create_comment:save_comment,1378,1178
selenium_create_comment,6284,6698
selenium_view_blog,9723,9937
selenium_view_dashboard,5365,5007
selenium_log_out,1543,1110
performance_profile.png contains the graph for the two test runs:
Scale Report
This report will show you how your Jira, Confluence or Bitbucket instance ran on 4, 2 and 1 node. Before generating this report you need to run all tests on 4, 2 and 1 node. After that you provide links to the three of the results in the scale_profile.yml file.
# Defines which column from test runs is used for aggregated report. Default is "90% Line"
column_name: "90% Line"
runs:
# fullPath should contain a full path to the directory with run results. E.g. /home/$USER/dc-app-performance-toolkit/jira/results/2019-08-06_18-41-08
- runName: "Node 1"
fullPath: "/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/1_node"
- runName: "Node 2"
fullPath: "/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/2_nodes"
- runName: "Node 4"
fullPath: "/Users/alexm/PycharmProjects/easymigration/dc-app-performance-toolkit/app/results/confluence/4_nodes"
# Chart generation configs
index_col: "Action"
title: "DCAPT Scale Testing"
image_height_px: 1000
image_width_px: 1200Again you will have three files: scale_profile_summary.log, scale_profile.cvs and scale_profile.png.
Those reports are supposed to be used for certifying your app for Data Center, but you can use the reports to get summary information about two or three test runs.
That is all for the reports and graphs. I believe, now you have enough information to start using dc-app-performance-toolkit.
Alexey Matveev 