Canonical Juju

removing and recreating a cmr relation fails

Bug #1727162 reported by Ian Booth
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
High
Andrew Wilkins
Canonical Juju 2.3-beta2

Bug Description

relate mediawiki to a hosted mysql
remove the relation
add the relation again

On the consuming model (mediawiki) the relation hooks fail to run and the logs contain uniter errors. It seems there's crud left behind from when the relation was broken when it was removed the first time.

unit-mediawiki-0: 14:29:44 INFO juju.worker.uniter hooks are retried true
unit-mediawiki-0: 14:29:44 INFO juju.worker.uniter committing "relation-broken" hook
unit-mediawiki-0: 14:29:44 ERROR juju.worker.uniter resolver loop error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
unit-mediawiki-0: 14:29:44 INFO juju.worker.uniter unit "mediawiki/0" shutting down: committing operation "skip run relation-broken (0) hook": unknown relation: 0
unit-mediawiki-0: 14:29:44 ERROR juju.worker.dependency "uniter" manifold worker returned unexpected error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
unit-mediawiki-0: 14:29:47 INFO worker.uniter.jujuc ensure jujuc symlinks in /var/lib/juju/tools/unit-mediawiki-0
unit-mediawiki-0: 14:29:47 INFO worker.uniter.jujuc was a symlink, now looking at /var/lib/juju/tools/2.3-beta2.1-trusty-amd64
unit-mediawiki-0: 14:29:47 INFO juju.worker.uniter unit "mediawiki/0" started
unit-mediawiki-0: 14:29:47 INFO juju.worker.uniter hooks are retried true
unit-mediawiki-0: 14:29:47 INFO juju.worker.uniter committing "relation-broken" hook
unit-mediawiki-0: 14:29:47 ERROR juju.worker.uniter resolver loop error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
unit-mediawiki-0: 14:29:47 INFO juju.worker.uniter unit "mediawiki/0" shutting down: committing operation "skip run relation-broken (0) hook": unknown relation: 0
unit-mediawiki-0: 14:29:47 ERROR juju.worker.dependency "uniter" manifold worker returned unexpected error: committing operation "skip run relation-broken (0) hook": unknown relation: 0
unit-mediawiki-0: 14:29:50 INFO worker.uniter.jujuc ensure jujuc symlinks in /var/lib/juju/tools/unit-mediawiki-0

Tags: cross-model
Ian Booth (wallyworld)
tags: added: cross-model
Andrew Wilkins (axwalk)
Changed in juju:
assignee: nobody → Andrew Wilkins (axwalk)
status: Triaged → In Progress
Revision history for this message
Andrew Wilkins (axwalk) wrote :

I had a bunch of issues when adding/removing/adding, eventually reproduce this one. The errors listed in the bug description were preceded with:

unit-mediawiki-0: 12:44:23 INFO juju.worker.uniter.operation skipped "db-relation-broken" hook (missing)
unit-mediawiki-0: 12:44:23 ERROR juju.worker.uniter resolver loop error: committing operation "run relation-broken (11) hook": permission denied
unit-mediawiki-0: 12:44:23 INFO juju.worker.uniter unit "mediawiki/0" shutting down: committing operation "run relation-broken (11) hook": permission denied
unit-mediawiki-0: 12:44:23 ERROR juju.worker.dependency "uniter" manifold worker returned unexpected error: committing operation "run relation-broken (11) hook": permission denied

AFAICS, that would be coming from the LeaveScope call.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

So, the issue is to do with relation unit refcounting: in EnterScope, remote units add to the refcount, but in some cases of LeaveScope, they do remove from it.

So what can happen is that the remote relation change saying the remote unit has departed is observed, and then reflected in the local model; that causes the relation to be removed, because of the refcounting mismatch. The local unit then departs and we go to remove it from scope, and find the relation is gone and get "permission denied".

We should be refcounting both local and remote units in all cases. How we choose to remove the relations is another matter. For example, on the consuming side we might say that the relation will be removed if it is Dying, and all local units leave scope; and on the offering side, when it's Dying and all consuming units have left scope.

Revision history for this message
Andrew Wilkins (axwalk) wrote :

https://github.com/juju/juju/pull/7996

Andrew Wilkins (axwalk)
Changed in juju:
status: In Progress → Fix Committed
Anastasia (anastasia-macmood)
Changed in juju:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.