senlin

Detach health_policy_event Policy failed in senlin project

Bug #1730295 reported by chenyb4 on 2017-11-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	senlin	Confirmed	Undecided	Duc Truong

Bug Description

Use senlin create cluste, and attach health_policy_event policy, but execute "senlin cluster-policy-detach -p pl_healthevent001 cl001" the error in log.

2017-11-06 02:23:41.943 3586 INFO senlin.engine.service [req-c94e5c1f-6c03-47dc-85ff-87f165e29c5e 0ed165520f9b443ca9040e4e1f44d7f0 631b9209a992429fb70909d810a4d3b9 - default default] Detaching policy 'pl_healthevent001' from cluster '7d296c35-0e2a-49f5-b86f-9666d34d84ba'.
2017-11-06 02:23:42.539 3586 INFO senlin.engine.service [req-c94e5c1f-6c03-47dc-85ff-87f165e29c5e 0ed165520f9b443ca9040e4e1f44d7f0 631b9209a992429fb70909d810a4d3b9 - - -] Policy detach action queued: b5fe5a85-7989-4e66-b4c3-7aff18dce23b.
2017-11-06 02:23:42.853 3586 INFO senlin.engine.event [req-c94e5c1f-6c03-47dc-85ff-87f165e29c5e 0ed165520f9b443ca9040e4e1f44d7f0 631b9209a992429fb70909d810a4d3b9 - - -] cl001 [7d296c35] CLUSTER_DETACH_POLICY - start: None
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server [req-903d23d1-2e01-41f4-a745-463e9afbef9a - - - - -] Exception during message handling: ValueError: list.remove(x): x not in list
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 442, in unregister_cluster
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server self._stop_check(entry)
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 346, in _stop_check
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server self.TG.thread_done(listener)
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 102, in thread_done
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server self.threads.remove(thread)
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server ValueError: list.remove(x): x not in list
2017-11-06 02:23:43.142 3586 ERROR oslo_messaging.rpc.server
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base [req-903d23d1-2e01-41f4-a745-463e9afbef9a - - - - -] Unexpected exception occurred during action CLUSTER_DETACH_POLICY (b5fe5a85-7989-4e66-b4c3-7aff18dce23b) execution: list.remove(x): x not in list
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
res = self.dispatcher.dispatch(message)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
result = func(ctxt, **new_args)

File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 442, in unregister_cluster
self._stop_check(entry)

File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 346, in _stop_check
self.TG.thread_done(listener)

File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 102, in thread_done
self.threads.remove(thread)

ValueError: list.remove(x): x not in list
: ValueError: list.remove(x): x not in list
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
res = self.dispatcher.dispatch(message)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
result = func(ctxt, **new_args)

File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 442, in unregister_cluster
self._stop_check(entry)

File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 346, in _stop_check
self.TG.thread_done(listener)

File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 102, in thread_done
self.threads.remove(thread)

ValueError: list.remove(x): x not in list
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base Traceback (most recent call last):
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/actions/base.py", line 491, in ActionProc
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base result, reason = action.execute()
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/actions/cluster_action.py", line 1038, in execute
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base res, reason = self._execute(**kwargs)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/actions/cluster_action.py", line 1007, in _execute
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base result, reason = method()
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base return f(*args, **kwargs)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/actions/cluster_action.py", line 918, in do_detach_policy
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base res, reason = self.entity.detach_policy(self.context, policy_id)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/cluster.py", line 398, in detach_policy
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base res, reason = policy.detach(self)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/policies/health_policy.py", line 213, in detach
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base health_manager.unregister(cluster.id)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 506, in unregister
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base return notify(engine_id, 'unregister_cluster', cluster_id=cluster_id)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 484, in notify
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base call_context.call(ctx, method, **kwargs)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base retry=self.retry)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base timeout=timeout, retry=retry)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base retry=retry)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base raise result
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base ValueError: list.remove(x): x not in list
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base Traceback (most recent call last):
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base res = self.dispatcher.dispatch(message)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base return self._do_dispatch(endpoint, method, ctxt, args)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base result = func(ctxt, **new_args)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 442, in unregister_cluster
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base self._stop_check(entry)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 346, in _stop_check
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base self.TG.thread_done(listener)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 102, in thread_done
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base self.threads.remove(thread)
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base ValueError: list.remove(x): x not in list
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.178 3586 ERROR senlin.engine.actions.base
2017-11-06 02:23:43.221 3586 ERROR senlin.engine.event [req-903d23d1-2e01-41f4-a745-463e9afbef9a - - - - -] cl001 [7d296c35] CLUSTER_DETACH_POLICY - error: list.remove(x): x not in list
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
res = self.dispatcher.dispatch(message)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
result = func(ctxt, **new_args)

File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 442, in unregister_cluster
self._stop_check(entry)

File "/usr/lib/python2.7/site-packages/senlin/engine/health_manager.py", line 346, in _stop_check
self.TG.thread_done(listener)

File "/usr/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 102, in thread_done
self.threads.remove(thread)

ValueError: list.remove(x): x not in list

The health_registry table data don't delete.
MariaDB [senlin]> select * from health_registry;
+--------------------------------------+--------------------------------------+------------------+----------+--------+--------------------------------------+---------+
| id | cluster_id | check_type | interval | params | engine_id | enabled |
+--------------------------------------+--------------------------------------+------------------+----------+--------+--------------------------------------+---------+
| c10554ca-1e5c-4753-bd65-a3e7387bbcc7 | 7d296c35-0e2a-49f5-b86f-9666d34d84ba | LIFECYCLE_EVENTS | 0 | {} | 5ac80d90-d71c-4997-9954-3604921bcff9 | 1 |
+--------------------------------------+--------------------------------------+------------------+----------+--------+--------------------------------------+---------+

The cluster_policy table data don't delete.

Revision history for this message

Duc Truong (dtruong) wrote on 2019-01-15:

Fixed in https://review.openstack.org/#/c/629689/

Changed in senlin:
status:	New → Fix Committed
status:	Fix Committed → Confirmed
assignee:	nobody → Duc Truong (dtruong)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.