Mirantis OpenStack

Enabling API rate limits with provided filter causes nova-api to fail

Bug #1683496 reported by Scott Machtmes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Fix Committed
High
Sergii Rizvan
Mirantis OpenStack 10.0
7.0.x
Fix Committed
High
Sergii Rizvan
Mirantis OpenStack 7.0-mu-9
8.0.x
Fix Released
High
Sergii Rizvan
Mirantis OpenStack 8.0-mu-5
9.x
Fix Released
High
Max Yatsenko
Mirantis OpenStack 9.2-mu-3

Bug Description

Environment = Mitaka MOS 9.2
Reproducible = yes

Enabling API rate limits by adding the ratelimit filter in the nova api-paste.ini configuration to the pipeline causes nova-api to produces errors and fail to serve api requests. This happens when using the default limits filter present since MOS 5.1 in the /etc/nova/api-paste.ini config.

The default filter example present is:

[filter:ratelimit]
limits=(POST, *, .*, 100000 , MINUTE); (POST, %(*/servers), ^/servers, 100000 , DAY);(PUT, %(*) , .*, 1000 , MINUTE);(GET, %(*changes-since*), .*changes-since.*, 100000, MINUTE);(DELETE, %(*), .*, 100000 , MINUTE)
paste.filter_factory=nova.api.openstack.compute.limits:RateLimitingMiddleware.factory

The syntax using % symbols causes errors upon restart of nova-api. Replacing the % and parentheses with double quotes allows the api to start without error. However, the proper syntax for deployment with puppet must be reviewed further to ensure correctness.

Errors seen using the provided values look like:

2017-04-17 17:57:50.101 18354 CRITICAL nova [-] InterpolationMissingOptionError: Error in file /etc/nova/api-paste.ini: Bad value substitution:
        section: [filter:ratelimit]
        option : limits
        key : */servers
        rawval : (POST, *, .*, 100000 , MINUTE); (POST, %(*/servers), ^/servers, 100000 , DAY);(PUT, %(*) , .*, 1000 , MINUTE);(GET, %(*changes-since*), .*changes-since.*, 100000, MINUTE);(DELETE, %(*), .*, 100000 , MINUTE)

2017-04-17 17:57:50.101 18354 ERROR nova Traceback (most recent call last):
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/bin/nova-api", line 10, in <module>
2017-04-17 17:57:50.101 18354 ERROR nova sys.exit(main())
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/api.py", line 57, in main
2017-04-17 17:57:50.101 18354 ERROR nova server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 360, in __init__
2017-04-17 17:57:50.101 18354 ERROR nova self.app = self.loader.load_app(name)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/wsgi.py", line 497, in load_app
2017-04-17 17:57:50.101 18354 ERROR nova return deploy.loadapp("config:%s" % self.config_path, name=name)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2017-04-17 17:57:50.101 18354 ERROR nova return loadobj(APP, uri, name=name, **kw)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2017-04-17 17:57:50.101 18354 ERROR nova return context.create()
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-04-17 17:57:50.101 18354 ERROR nova return self.object_type.invoke(self)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-04-17 17:57:50.101 18354 ERROR nova **context.local_conf)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-17 17:57:50.101 18354 ERROR nova val = callable(*args, **kw)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/api/openstack/urlmap.py", line 160, in urlmap_factory
2017-04-17 17:57:50.101 18354 ERROR nova app = loader.get_app(app_name, global_conf=global_conf)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-04-17 17:57:50.101 18354 ERROR nova name=name, global_conf=global_conf).create()
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, i
n create
2017-04-17 17:57:50.101 18354 ERROR nova return self.object_type.invoke(self)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-04-17 17:57:50.101 18354 ERROR nova **context.local_conf)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2017-04-17 17:57:50.101 18354 ERROR nova val = callable(*args, **kw)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/api/auth.py", line 79, in pipeline_factory_v21
2017-04-17 17:57:50.101 18354 ERROR nova return _load_pipeline(loader, local_conf[CONF.auth_strategy].split())
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/api/auth.py", line 58, in _load_pipeline
2017-04-17 17:57:50.101 18354 ERROR nova filters = [loader.get_filter(n) for n in pipeline[:-1]]
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 354, in get_filter
2017-04-17 17:57:50.101 18354 ERROR nova name=name, global_conf=global_conf).create()
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 366, in filter_context
2017-04-17 17:57:50.101 18354 ERROR nova FILTER, name=name, global_conf=global_conf)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 430, in get_context
2017-04-17 17:57:50.101 18354 ERROR nova local_conf[option] = self.parser.get(section, option)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/ConfigParser.py", line 623, in get
2017-04-17 17:57:50.101 18354 ERROR nova return self._interpolate(section, option, value, d)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 75, in _interpolate
2017-04-17 17:57:50.101 18354 ERROR nova self, section, option, rawval, vars)
2017-04-17 17:57:50.101 18354 ERROR nova File "/usr/lib/python2.7/ConfigParser.py", line 669, in _interpolate
2017-04-17 17:57:50.101 18354 ERROR nova option, section, rawval, e.args[0])
2017-04-17 17:57:50.101 18354 ERROR nova InterpolationMissingOptionError: Error in file /etc/nova/api-paste.ini: Bad value substitution:
2017-04-17 17:57:50.101 18354 ERROR nova section: [filter:ratelimit]
2017-04-17 17:57:50.101 18354 ERROR nova option : limits
2017-04-17 17:57:50.101 18354 ERROR nova key : */servers
2017-04-17 17:57:50.101 18354 ERROR nova rawval : (POST, *, .*, 100000 , MINUTE); (POST, %(*/servers), ^/servers, 100000 , DAY);(PUT, %(*) , .*, 1000 , MINUTE);(GET, %(*changes-since*), .*changes-since.*, 100000, MINUTE);(DELETE, %(*), .*, 100000 , MINUTE)
2017-04-17 17:57:50.101 18354 ERROR nova
2017-04-17 17:57:50.101 18354 ERROR nova

Denis Meltsaykin (dmeltsaykin)
Changed in mos:
milestone: none → 9.x-updates
assignee: nobody → MOS Maintenance (mos-maintenance)
importance: Undecided → High
status: New → Confirmed
Sergii Rizvan (srizvan)
Changed in mos:
assignee: MOS Maintenance (mos-maintenance) → Sergii Rizvan (srizvan)
Sergii Rizvan (srizvan)
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Sergii Rizvan (srizvan) wrote :

Note for QA engineers:
Before reproducing the bug, API rate limits should be enabled by adding the ratelimit filter in [composite:openstack_compute_api_v21] section (for mitaka) in the nova api-paste.ini:

...
[composite:openstack_compute_api_v21]
use = call:nova.api.auth:pipeline_factory_v21
noauth2 = cors compute_req_id faultwrap sizelimit noauth2 osapi_compute_app_v21
keystone = cors compute_req_id faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v21
...

For outher versions 'ratelimit' filter may need additions to other sections of the nova api-paste.ini

Revision history for this message
Sergii Rizvan (srizvan) wrote :

Patch is on review: https://review.openstack.org/#/q/I96c00590390c6caf0a17a43e68ede111684b3110,n,z

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on Fuel 8.0 MU-5:

root@node-3:~# tail -n 4 /etc/nova/api-paste.ini

[filter:ratelimit]
limits=(POST, *, .*, 100000 , MINUTE);(POST, "*/servers", ^/servers, 100000 , DAY);(PUT, "*" , .*, 1000 , MINUTE);(GET, "*changes-since*", .*changes-since.*, 100000, MINUTE);(DELETE, "*", .*, 100000 , MINUTE)
paste.filter_factory=nova.api.openstack.compute.limits:RateLimitingMiddleware.factory

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/36844

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/fuel-library (stable/mitaka)

Change abandoned by Max Yatsenko <email address hidden> on branch: stable/mitaka
Review: https://review.fuel-infra.org/36844

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/36846

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/fuel-library (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/36846
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: f683effc06211bb9cabd9e00c856121a15706ef5
Author: Sergii Rizvan <email address hidden>
Date: Thu Oct 12 07:56:01 2017

Corrected default limits filter in nova API

The old syntax that used % symbols, caused some errors upon a restart
of the nova-api. Replacing the % and parentheses with the double quotes
allows the API to start without errors.

Change-Id: I96c00590390c6caf0a17a43e68ede111684b3110
Closes-Bug: #1683496

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

Verified on Fuel 9.2 MU-3 (snapshots/9.0-2017-10-16-142324):

root@node-3:~# tail -n 4 /etc/nova/api-paste.ini

[filter:ratelimit]
limits=(POST, *, .*, 100000 , MINUTE); (POST, "*/servers", ^/servers, 100000 , DAY);(PUT, "*" , .*, 1000 , MINUTE);(GET, "*changes-since*", .*changes-since.*, 100000, MINUTE);(DELETE, "*", .*, 100000 , MINUTE)
paste.filter_factory=nova.api.openstack.compute.limits:RateLimitingMiddleware.factory

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library ocata-eol

This issue was fixed in the openstack/fuel-library ocata-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.