Mirantis OpenStack

Performance degradation in Keystone after 663th build in 9.2

Bug #1652950 reported by Pavel
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Pavel
Mirantis OpenStack 9.2

Bug Description

Looks like we got new significant performance degradation in Keystone.

It has definitily occured after 663-th build because previous performance results we got exactly for this build https://mirantis.testrail.com/index.php?/runs/view/28558
And they were wery close to results for 9.1 (except not so huge 50-th percentile degradation for some of tests)

And here are the results for 689th build where significant performance degradation was found:
https://mirantis.testrail.com/index.php?/runs/view/29112

As you can see, we got about 100% performance degradation for Combined Tests (get token then validate it) comparatively to performance baselines in 9.1 MU.

Both test runs were performed using the same hardware environment (ENV-18 Lab of Performance QA Team). Installation procedure and cluster parameters were the same except using OVS FW in 689th instead of Iptables WF in 663th.

Tatyana Borukhovich (tatyanaborukhovich)
tags: added: blocker-for-qa
Revision history for this message
Boris Bobrov (bbobrov) wrote :

Nothing has changed in keystone during the last couple of months, so probably something else caused this. Could you please bisect the snapshots and find the first bad one?

Changed in mos:
assignee: nobody → Pavel (p.petrov)
Revision history for this message
Pavel (p.petrov) wrote :

I started to bisect yet at 29.12.2016 having only 1 ready environment (18-th in PerfQA-Lab).
Then at some moment I got another one and decided to continue bisecting on that environment whilst it was planned to use 18-th env to repeat tests for the 689-th build.

I want to remind that for the first time we faced degradation for short run and long run:
https://mirantis.testrail.com/index.php?/runs/view/29118 (short)
https://mirantis.testrail.com/index.php?/runs/view/29118 (long)

This time results showed that on the same (18-th) environment and in the same cluster configuration 90-th percentille degradation wasn't reproduced neither after short nor after long run:
https://mirantis.testrail.com/index.php?/runs/view/29650 (short)
https://mirantis.testrail.com/index.php?/runs/view/29657 (long)

Please, don't mind on about 20% degradation for 50-th percentille - it is caused by 18-th environment itself. I've repeated tests for 689-th build on the reference environment (14-th in PerfQA-Lab) and results showed no significant degradation neither for 50-th nor for 90-th percentille.

Also, I performed tests for 718-th build on that reference environment and the results are acceptable: https://mirantis.testrail.com/index.php?/runs/view/29656

Vitaly Sedelnik (vsedelnik)
Changed in mos:
milestone: none → 9.2
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Sergii (sgudz) wrote :

Verified on 750 snapshot. Not reproduced

Changed in mos:
status: Confirmed → Invalid
Revision history for this message
Sergii (sgudz) wrote :

Results are acceptable - https://mirantis.testrail.com/index.php?/runs/view/30399&group_by=cases:section_id&group_order=asc

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.