neutron

cannot update qos rule

Bug #1815618 reported by Bence Romsics on 2019-02-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Lajos Katona	neutron stein-rc1

Bug Description

This bug seems to be combination of problems on both client and server sides. So we may need to add pyhton-neutronclient and/or python-openstackclient as an affected component. I'll do that as soon as I manage to locate which one contains the client side bug. But this report will be good to track the overall problem.

First the reproduction:

openstack network qos policy create policy0

openstack network qos rule create policy0 --type minimum-bandwidth --min-kbps 1000 --egress # 71a84995-cccd-4f09-9c3d-b1caa18ff363
openstack network qos rule set policy0 71a84995-cccd-4f09-9c3d-b1caa18ff363 --min-kbps 1001 --egress
-> works as expected

# make sure we only have one rule of the type
openstack network qos rule delete policy0 71a84995-cccd-4f09-9c3d-b1caa18ff363

openstack network qos rule create policy0 --type minimum-bandwidth --min-kbps 1000 --ingress # 1155c1c8-f9a7-4954-b195-9f58c8e18b4d
openstack network qos rule set policy0 1155c1c8-f9a7-4954-b195-9f58c8e18b4d --min-kbps 1001 --ingress
-> works as expected

openstack network qos rule delete policy0 1155c1c8-f9a7-4954-b195-9f58c8e18b4d

# create the ingress/egress pair at once
openstack network qos rule create policy0 --type minimum-bandwidth --min-kbps 1000 --egress # f392837a-09e2-4b5e-8c29-86670797679e
openstack network qos rule create policy0 --type minimum-bandwidth --min-kbps 1000 --ingress # 77dae223-b787-4943-bb45-c42424fd29ec

# This is the bug. As we'll see later the trigger is a client-side problem, but I don't think neutron-server should return 500 Internal Server Error. The malformed input should be caught earlier and a 4xx response should be given.
openstack network qos rule set policy0 f392837a-09e2-4b5e-8c29-86670797679e --min-kbps 1001 --egress
Failed to set Network QoS rule ID "f392837a-09e2-4b5e-8c29-86670797679e": HttpException: 500: Server Error for url: http://100.109.0.20:9696/v2.0/qos/policies/188a2f59-ab90-41a3-9e6f-58e641a34544/minimum_bandwidth_rules/f392837a-09e2-4b5e-8c29-86670797679e, Request Failed: internal server error while processing your request.

openstack network qos rule set policy0 77dae223-b787-4943-bb45-c42424fd29ec --min-kbps 1001 --ingress
Failed to set Network QoS rule ID "77dae223-b787-4943-bb45-c42424fd29ec": HttpException: 500: Server Error for url: http://100.109.0.20:9696/v2.0/qos/policies/188a2f59-ab90-41a3-9e6f-58e641a34544/minimum_bandwidth_rules/77dae223-b787-4943-bb45-c42424fd29ec, Request Failed: internal server error while processing your request.

# the same rule update can be done by neutronclient, but only for the egress direction
neutron qos-minimum-bandwidth-rule-update f392837a-09e2-4b5e-8c29-86670797679e policy0 --min-kbps 1001 --direction egress
-> works as expected

# this failure is expected because neutronclient was long deprecated already when the ingress direction was introduced
neutron qos-minimum-bandwidth-rule-update 77dae223-b787-4943-bb45-c42424fd29ec policy0 --min-kbps 1001 --direction ingress
neutron qos-minimum-bandwidth-rule-update: error: argument --direction: invalid choice: u'ingress' (choose from 'egress')

Further details:

The working update by neutronclient looks like this:

neutron qos-minimum-bandwidth-rule-update f392837a-09e2-4b5e-8c29-86670797679e policy0 --min-kbps 1001 --direction egress
PUT /v2.0/qos/policies/188a2f59-ab90-41a3-9e6f-58e641a34544/minimum_bandwidth_rules/f392837a-09e2-4b5e-8c29-86670797679e
{"minimum_bandwidth_rule": {"direction": "egress", "min_kbps": "1001"}}

The wrong update by openstackclient looks like this:

openstack network qos rule set policy0 f392837a-09e2-4b5e-8c29-86670797679e --min-kbps 1001 --egress
PUT /v2.0/qos/policies/188a2f59-ab90-41a3-9e6f-58e641a34544/minimum_bandwidth_rules/f392837a-09e2-4b5e-8c29-86670797679e
{"minimum_bandwidth_rule": {"min_kbps": 1001}}

The 500 Internal Server Error's traceback:

febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource [None req-e73566b0-35e0-4572-a3d5-28f77853b4c9 admin admin] update failed: No details.: AttributeError: 'QosMinimumBandwidthRule' object has no attribute '_ob
j_direction'
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource Traceback (most recent call last):
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource result = method(request=request, **args)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 624, in update
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource return self._update(request, id, body, **kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron-lib/neutron_lib/db/api.py", line 142, in wrapped
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource self.force_reraise()
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron-lib/neutron_lib/db/api.py", line 138, in wrapped
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 154, in wrapper
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource self.force_reraise()
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 142, in wrapper
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron-lib/neutron_lib/db/api.py", line 198, in wrapped
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource self.force_reraise()
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron-lib/neutron_lib/db/api.py", line 194, in wrapped
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 680, in _update
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/extensions/qos.py", line 174, in <lambda>
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource method_name, rule_cls, *args, **kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/extensions/qos.py", line 170, in _make_call
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource context, rule_cls, *args_list, **params
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 50, in inner
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource result = f(*args, **kwargs)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/services/qos/qos_plugin.py", line 423, in update_policy_rule
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource checker.check_rules_conflict(policy, rule)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/objects/qos/qos_policy_validator.py", line 63, in check_rules_conflict
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource if rule.duplicates(rule_obj):
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/objects/qos/rule.py", line 83, in duplicates
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource if getattr(self, field) != getattr(other_rule, field):
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 68, in getter
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource return getattr(self, attrname)
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource AttributeError: 'QosMinimumBandwidthRule' object has no attribute '_obj_direction'
febr 12 12:20:19 devstack0 neutron-server[31565]: ERROR neutron.api.v2.resource·

The version used to reproduce the bug:

neutron 2f3cc51784
neutron-lib aceb7c50ed
devstack ee4b6a01
python-openstackclient dcff1012fd
python-neutronclient d74b871f7fe
openstacksdk==0.23.0
osc-lib==1.12.0

I'll work on fixing these problems.

Tags:

Brian Haley (brian-haley) on 2019-02-12

Changed in neutron:
importance:	Undecided → High
status:	New → Confirmed

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2019-02-14:

Quick summary of my findings so far:

At the time of reporting I thought this is a combination of client and server side bugs. But now I think this should be fixed purely on server side.

My original thinking was based on the fact that neutronclient can update a qos rule, while osc cannot. And my first attempt for a fix was to get osc to send the same body as neutronclient does.

But if you take a closer look at the PUT bodies sent by both clients, you may ask why neutron-server mandates the presence of the direction field in the body even when the direction field is not being updated? We only want to update min_kbps here. So the body sent by osc should be accepted by neutron-server as a valid request.

While debugging I also realized it's practically impossible to get osc (ie. openstacksdk more precisely) to send the same body as neutronclient does. The basic reason is the logic in the _update() method here:

https://github.com/openstack/openstacksdk/blob/master/openstack/proxy.py#L178

Instead of sending a single PUT, it first GETs the resource and only includes the changed fields into the PUT body. By that it does not care if '[--ingress|--egress]' was present on the command line as long as the direction field's value does not change. In the source I don't see a way to force the inclusion a field that didn't change. So with the current openstacksdk I don't think this can be fixed on the client side.

Anyway there's an easy workaround if someone is willing to fiddle with curl:

$ cat min-bw-rule-update
#! /bin/sh

# export TOKEN="$( openstack token issue -f value -c id )"
# min-bw-rule-update NEUTRON-BASE-URL POLICY-ID RULE-ID DIRECTION MIN-KBPS
# min-bw-rule-update http://127.0.0.1:9696 188a2f59-ab90-41a3-9e6f-58e641a34544 f392837a-09e2-4b5e-8c29-86670797679e egress 1001

neutron_base_url="${1:?}"
policy_id="${2:?}"
rule_id="${3:?}"
direction="${4:?}"
min_kbps="${5:?}"

cat <<EOF |
{"minimum_bandwidth_rule":
    {"direction": "$direction",
     "min_kbps": "$min_kbps"}}
EOF
curl \
    --silent \
    --insecure \
    --request PUT \
    --header "content-type: application/json" \
    --header "x-auth-token: ${TOKEN:?}" \
    --data @- \
    "${neutron_base_url:?}/v2.0/qos/policies/$policy_id/minimum_bandwidth_rules/$rule_id" \
| json_pp

lajoskatona told me he's looking into the server side of this bug, the next update may be coming from him.

Quick summary of my findings so far:

At the time of reporting I thought this is a combination of client and server side bugs. But now I think this should be fixed purely on server side.

My original thinking was based on the fact that neutronclient can update a qos rule, while osc cannot. And my first attempt for a fix was to get osc to send the same body as neutronclient does.

https://github.com/openstack/openstacksdk/blob/master/openstack/proxy.py#L178

Anyway there's an easy workaround if someone is willing to fiddle with curl:

$ cat min-bw-rule-update 
#! /bin/sh

neutron_base_url="${1:?}"
policy_id="${2:?}"
rule_id="${3:?}"
direction="${4:?}"
min_kbps="${5:?}"

lajoskatona told me he's looking into the server side of this bug, the next update may be coming from him.

Lajos Katona (lajos-katona) on 2019-02-19

Changed in neutron:
assignee:	Bence Romsics (bence-romsics) → Lajos Katona (lajos-katona)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-02-22: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/638714

Changed in neutron:
status:	Confirmed → In Progress

Akihiro Motoki (amotoki) on 2019-03-12

Changed in neutron:
milestone:	none → stein-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-12: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/638714
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c1999a2b2268a4f5477247974363e31cb54e0e6b
Submitter: Zuul
Branch: master

commit c1999a2b2268a4f5477247974363e31cb54e0e6b
Author: Lajos Katona <email address hidden>
Date: Fri Mar 1 14:39:13 2019 +0100

Fix QoS rule update

    QoS rule (QosBandwidthLimitRule and QosMinimumBandwidthRule) update now
    expects to have direction field in the API request for checking for
    duplicates.
    This patch changes this by using the rule fetched from the policy and
    the update will work on that rule object instead of a newly created
    object which for minimum_bandwidth rule has no default direction, which
    causes the update to fail.

Change-Id: Ib8f95bf14193a50f22102668bed9208a93d1caba
Closes-Bug: #1815618

Changed in neutron:
status:	In Progress → Fix Released

Bernard Cafarelli (bcafarel) on 2019-03-15

tags:

added: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-21: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/645172

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-24: Fix included in openstack/neutron 14.0.0.0rc1

This issue was fixed in the openstack/neutron 14.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-28: Fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/645172
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e7788ed0c80ba5f45eafeda5a6c07a8300156535
Submitter: Zuul
Branch: stable/rocky

commit e7788ed0c80ba5f45eafeda5a6c07a8300156535
Author: Lajos Katona <email address hidden>
Date: Fri Mar 1 14:39:13 2019 +0100

Fix QoS rule update

    Change-Id: Ib8f95bf14193a50f22102668bed9208a93d1caba
    Closes-Bug: #1815618
    (cherry picked from commit c1999a2b2268a4f5477247974363e31cb54e0e6b)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-29: Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/648648

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-05: Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/650258

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-06: Fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/648648
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5a11b24f370a122f56a2bbf622824caaff73aa78
Submitter: Zuul
Branch: stable/queens

commit 5a11b24f370a122f56a2bbf622824caaff73aa78
Author: Lajos Katona <email address hidden>
Date: Fri Mar 1 14:39:13 2019 +0100

Fix QoS rule update

    Change-Id: Ib8f95bf14193a50f22102668bed9208a93d1caba
    Closes-Bug: #1815618
    (cherry picked from commit c1999a2b2268a4f5477247974363e31cb54e0e6b)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-07: Fix merged to neutron (stable/pike)

#10

Reviewed: https://review.openstack.org/650258
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff0a4445267da908376d14d1b4fa398f2ecfb2b8
Submitter: Zuul
Branch: stable/pike

commit ff0a4445267da908376d14d1b4fa398f2ecfb2b8
Author: Lajos Katona <email address hidden>
Date: Fri Mar 1 14:39:13 2019 +0100

Fix QoS rule update

    Change-Id: Ib8f95bf14193a50f22102668bed9208a93d1caba
    Closes-Bug: #1815618
    (cherry picked from commit c1999a2b2268a4f5477247974363e31cb54e0e6b)