Cannot establish CMR - macaroon auth fails

Bug #2021969 reported by Vitaly Antonenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Committed
Critical
Ian Booth

Bug Description

Since the release of Juju version 3.1, I have encountered a critical problem where I am unable to establish cross-model relations.

Steps to Reproduce:

- Install and configure Juju version 3.1 on a supported environment.
- Create multiple models within Juju.
- Attempt to establish a cross-model relation between any two models.
- Observe the error or unexpected behavior preventing the successful establishment of the cross-model relation.

Short steps to reproduce:
```
cd tests
./main.sh -v -p lxd cmr run_offer_consume
```

Detail steps:
```
juju add-model model-offer

echo "Deploy consumed workload and create the offer"
juju deploy juju-qa-dummy-source
juju offer dummy-source:sink dummy-offer

echo "Check list-offer output"
juju list-offers --format=json | jq -r 'has("dummy-offer")' | check true

echo "Deploy workload in consume model"
juju add-model "model-consume"
juju switch "model-consume"
juju deploy juju-qa-dummy-sink

echo "Relate workload in consume model with offer"
juju consume "admin/model-offer.dummy-offer"
juju relate dummy-sink dummy-offer

echo "Provide config for offered workload and change the status of consumed offer"
# Change the dummy-source config for "token" and check that the change
# is represented in the consuming model's dummy-sink unit.
juju switch "model-offer"
juju config dummy-source token=yeah-boi
juju switch "model-consume"

juju status
# wait for the offer goes to an error state

```

It should be possible to establish cross-model relations by specifying the appropriate relation endpoints.

Revision history for this message
Ian Booth (wallyworld) wrote :

Juju 3.1.2 works fine for me running a test on aws using the dummy sink and source charms.

Just because the consume model shows a saas error, that is not necessarily a cmr issue - all that indicates is that the offered application has gone into error and that is reported in the consumer model.

You need to investigate the underlying reason why the offered app has gone into error, or if not, what the logs etc show.

Changed in juju:
status: New → Incomplete
Revision history for this message
Jack Shaw (jack-shaw) wrote :

I've managed to replicate this on juju 3.1.3, using lxd and aws, and the same dummy charms

Logs show:
```
controller-0: 12:32:50 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:32:50 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:32:50 ERROR juju.worker.remoterelations cmr error in relation units worker for dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:32:50 ERROR juju.worker.remoterelations cmr error in relation units worker for dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:32:50 ERROR juju.worker.remoterelations cmr error in remote application worker for dummy-offer: handling change for relation "dummy-offer:sink dummy-sink:source": watching consumed secret changes: verification failed: missing offer

controller-0: 12:33:05 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:33:05 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:33:05 ERROR juju.worker.remoterelations cmr error in relation units worker for dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:33:05 ERROR juju.worker.remoterelations cmr error in relation units worker for dummy-offer:sink dummy-sink:source: connection is shut down
controller-0: 12:33:05 ERROR juju.worker.remoterelations cmr error in remote application worker for dummy-offer: handling change for relation "dummy-offer:sink dummy-sink:source": watching consumed secret changes: verification failed: missing offer
```

Logs from the source model have nothing of note that I could see

Revision history for this message
Jack Shaw (jack-shaw) wrote (last edit ):

I have also been able to replicate something similar with easyrsa and etcd, although the logs are slightly different

```
controller-0: 13:40:17 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:17 ERROR juju.worker.remoterelations cmr error in relation units worker for etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:17 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:17 ERROR juju.worker.remoterelations cmr error in relation units worker for etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:17 ERROR juju.worker.remoterelations cmr error in remote application worker for easyrsa: handling change for relation "etcd:certificates easyrsa:client": watching consumed secret changes: permission denied

controller-0: 13:40:32 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:32 ERROR juju.worker.remoterelations cmr error in relation units worker for etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:32 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:40:32 ERROR juju.worker.remoterelations cmr error in relation units worker for etcd:certificates easyrsa:client: connection is shut down
controller-0: 13:42:33 ERROR juju.worker.remoterelations cmr error in remote application worker for easyrsa: handling change for relation "etcd:certificates easyrsa:client": watching consumed secret changes: verification failed: missing offer
...
```

But perhaps this is a separate issue?

Revision history for this message
Max Asnaashari (masnax) wrote :

This is also happening for me on 3.1/edge (so 3.1.3) for lxd-cloud and lxd. I'm unable to reproduce it on 3.1/stable (3.1.2).

From `juju debug-log` I get
```
controller-0: 18:27:49 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation lxd-cloud-cell:https virtual-lxd:https: connection is shut down
controller-0: 18:27:49 ERROR juju.worker.remoterelations cmr error in relation units worker for lxd-cloud-cell:https virtual-lxd:https: connection is shut down
controller-0: 18:27:49 ERROR juju.worker.remoterelations cmr error in remote relations worker for relation lxd-cloud-cell:https virtual-lxd:https: connection is shut down
controller-0: 18:27:49 ERROR juju.worker.remoterelations cmr error in relation units worker for lxd-cloud-cell:https virtual-lxd:https: connection is shut down
controller-0: 18:27:49 ERROR juju.worker.remoterelations cmr error in remote application worker for virtual-lxd: handling change for relation "lxd-cloud-cell:https virtual-lxd:https": watching consumed secret changes: permission denied
```

Changed in juju:
status: Incomplete → Confirmed
Revision history for this message
Ian Booth (wallyworld) wrote :

It works in 3.1.2 but does break on edge.

The issue is that the macaroon used for auth is missing the offer-uuid first party caveat.

summary: - Cannot establish CMR staring from 3.1
+ Cannot establish CMR - macaroon auth fails
Changed in juju:
importance: Undecided → Critical
status: Confirmed → Triaged
status: Triaged → In Progress
assignee: nobody → Ian Booth (wallyworld)
Revision history for this message
Ian Booth (wallyworld) wrote :
Changed in juju:
milestone: none → 3.1.3
Ian Booth (wallyworld)
Changed in juju:
status: In Progress → Fix Committed
milestone: 3.1.3 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.