HA test failed because the request that was made requires authentication. (HTTP 401)

Bug #1778103 reported by Raoul Scarazzini
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Won't Fix
High
Unassigned

Bug Description

While testing HA in RDOPhase2 we do a test named "pacemaker-light-a". We stop all the systemd resources managed by Pacemaker (so just openstack-cinder-volume), then we stop overcloud's core resources (Galera, Rabbit, and Redis) and then we start openstack-cinder-volume again, checking that we don't have failed actions on the cluster.
After the recovery of the core resources, so again after verifying that we don't have failed actions on the cluster side, we start an openstack-heat template [1] that spawns an instance using a previously created volume.
We hit this problem while creating the stack:

2018-06-20 21:37:21Z [stack_test_pacemaker-light-a.instance]: CREATE_FAILED Unauthorized: resources.instance: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9972ab52-4b39-4a34-88e3-9b6e20a8d149)",
2018-06-20 21:37:23Z [stack_test_pacemaker-light-a]: CREATE_FAILED Resource CREATE failed: Unauthorized: resources.instance: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9972ab52-4b39-4a34-88e3-9b6e20a8d149)

The error seems to be related to the authentication layer. Looking at the heat logs [2] the full error is:

2018-06-20 21:37:21.723 7 ERROR heat.engine.resource Traceback (most recent call last):
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 919, in _action_recorder
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource yield
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1027, in _do_action
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource step = next(subtask)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 978, in action_handler_task
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource done = check(handler_data)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 872, in check_create_complete
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource check = self.client_plugin()._check_active(server_id)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 221, in _check_active
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource server = self.fetch_server(server)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/nova.py", line 150, in fetch_server
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource server = self.client().servers.get(server_id)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 811, in get
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource return self._get("/servers/%s" % base.getid(server), "server")
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 351, in _get
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource resp, body = self.api.client.get(url)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource return self.request(url, 'GET', **kwargs)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource raise exceptions.from_response(resp, body, url, method)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9972ab52-4b39-4a34-88e3-9b6e20a8d149)
2018-06-20 21:37:21.723 7 ERROR heat.engine.resource

But there seem not to be an explicit error inside keystone log [3], so it might be related to some other component.

[1] https://github.com/openstack/tripleo-ha-utils/blob/master/roles/validate-ha/templates/validate-ha-heat-template.yaml.j2
[2] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-174/overcloud-controller-0/var/log/containers/heat/heat_api.log.txt.gz
[3] https://thirdparty.logs.rdoproject.org/jenkins-oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans-174/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz

Tags: ci
Raoul Scarazzini (rasca)
Changed in tripleo:
importance: Undecided → High
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Revision history for this message
Gabriele Cerami (gcerami) wrote :

Everything has changed in this bug since its last meaningful update. Raoul has moved to other positions, rdophase2 is not in our scope anymore. MOving to WOntfix

Changed in tripleo:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.