OpenStack Glance-Simplestreams-Sync Charm

(cert cache v3 issues?) hook failed: "certificates-relation-changed" / AttributeError: 'NoneType' object has no attribute 'encode'

Bug #2017671 reported by Dominik Bender
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Glance-Simplestreams-Sync Charm
Invalid
Undecided
Unassigned
vault-charm
Incomplete
Undecided
Unassigned

Bug Description

After deployment of glance-simplestreams-sync application in a existing openstack bundle the charm stuck in error hook failed: "certificates-relation-changed".

Model Controller Cloud/Region Version SLA Timestamp
dbi7-c1 dbi7-prod dbi7/default 3.1.2 unsupported 12:57:33Z

App Version Status Scale Charm Channel Rev Exposed Message
glance-simplestreams-sync error 1 glance-simplestreams-sync zed/stable 92 no hook failed: "certificates-relation-changed"
octavia-diskimage-retrofit 1.0.1 active 1 octavia-diskimage-retrofit zed/stable 93 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
glance-simplestreams-sync/5* error idle 2/lxd/47 10.105.121.46 hook failed: "certificates-relation-changed"
  octavia-diskimage-retrofit/5* active idle 10.105.121.46 Unit is ready

I get this debug-log:
---
unit-glance-simplestreams-sync-5: 12:24:01 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-glance-simplestreams-sync-5: 12:24:01 INFO unit.glance-simplestreams-sync/5.juju-log certificates:465: Making dir /etc/apache2/ssl/glance-simplestreams-sync root:root 555
unit-glance-simplestreams-sync-5: 12:24:01 INFO unit.glance-simplestreams-sync/5.juju-log certificates:465: Installing CA certificate from certificate relation
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed Traceback (most recent call last):
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed File "/var/lib/juju/agents/unit-glance-simplestreams-sync-5/charm/hooks/certificates-relation-changed", line 437, in <module>
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed hooks.execute(sys.argv)
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed File "/var/lib/juju/agents/unit-glance-simplestreams-sync-5/charm/charmhelpers/core/hookenv.py", line 963, in execute
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed self._hooks[hook_name]()
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed File "/var/lib/juju/agents/unit-glance-simplestreams-sync-5/charm/hooks/certificates-relation-changed", line 416, in certs_changed
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed process_certificates('glance-simplestreams-sync', relation_id, unit)
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed File "/var/lib/juju/agents/unit-glance-simplestreams-sync-5/charm/charmhelpers/contrib/openstack/cert_utils.py", line 390, in process_certificates
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed _manage_ca_certs(ca, relation_id)
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed File "/var/lib/juju/agents/unit-glance-simplestreams-sync-5/charm/charmhelpers/contrib/openstack/cert_utils.py", line 351, in _manage_ca_certs
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed ca.encode(),
unit-glance-simplestreams-sync-5: 12:24:02 WARNING unit.glance-simplestreams-sync/5.certificates-relation-changed AttributeError: 'NoneType' object has no attribute 'encode'
unit-glance-simplestreams-sync-5: 12:24:02 ERROR juju.worker.uniter.operation hook "certificates-relation-changed" (via explicit, bespoke hook script) failed: exit status 1
unit-glance-simplestreams-sync-5: 12:24:02 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
---

Remove und readd the certificates relation doesn't help.

Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote (last edit ):

That's a curious bug. It probably means that the ca is empty, but there are certificates. Could you do a juju show-unit on the glance-ss unit and the vault leader to see what's on the relations. Thanks. Also could you provide the juju status output for the model.

Changed in charm-glance-simplestreams-sync:
status: New → Incomplete
Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Yup, that doesn't look at all correct:

  - relation-id: 466
    endpoint: certificates
    related-endpoint: certificates
    application-data: {}
    related-units:
      vault/0:
        in-scope: true
        data:
          client.cert: |-
            -----BEGIN CERTIFICATE-----
            MIID...lWC1h6
            -----END CERTIFICATE-----
          client.key: |-
            -----BEGIN RSA PRIVATE KEY-----
            MIIE...Zvzn0g=
            -----END RSA PRIVATE KEY-----
          egress-subnets: 10.105.121.5/32
          glance-simplestreams-sync_5.processed_requests: '{"juju-b9f7b8-2-lxd-47.de1.dbi7.net":
            {"cert": "-----BEGIN CERTIFICATE-----\nMIIE...C1/kZs=\n-----END
            CERTIFICATE-----", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIE...Pxxlwk\n-----END
            RSA PRIVATE KEY-----"}}'
          ingress-address: 10.105.121.5
          private-address: 10.105.121.5

If you know how to do a crashdump, then that would be great, otherwise the complete juju logs from glance-ss and the vault units would be handy. Also, there only appear to be two vault units in ha, which is a bit strange?

Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Dominik Bender (ephermeral) wrote :

I had 3 units before. After an upgrade from juju 2.9 -> 3.1 the unit vault/2 had a problem and I deployed the unit again. Unfortunately, the unit could not join the cluster (https://bugs.launchpad.net/vault-charm/+bug/2017514). I have decided to remove the unit first until there is a solution.

Possibly the problems are related or have the same cause.

Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Dominik Bender (ephermeral) wrote :
Revision history for this message
Dominik Bender (ephermeral) wrote (last edit ):

You're right. There should be the ca.

Here the output from the vault leader unit with the part of relation-id: 466:

- relation-id: 466
    endpoint: certificates
    related-endpoint: certificates
    application-data: {}
    related-units:
      glance-simplestreams-sync/5:
        in-scope: true
        data:
          cert_requests: '{"juju-b9f7b8-2-lxd-47.de1.dbi7.net": {"sans": ["10.105.121.46"]}}'
          egress-subnets: 10.105.121.46/32
          ingress-address: 10.105.121.46
          private-address: 10.105.121.46
          unit_name: glance-simplestreams-sync_5

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

@ephemeral (Dominik) I'm wondering if it was actually a vault issue (that we recently reverted) that might have been causing the issue. If it's possible, could you retest with latest vault. If you need to upgrade charm, then please pause all the vault units prior to doing the upgrade charm/refresh command as otherwise the vault units will go into error. If you can't retest, that's okay; I'm going to mark this invalid on gss, and add it to vault as a new data point for vault.

Changed in charm-glance-simplestreams-sync:
status: Incomplete → Invalid
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Added to vault; I think this is part of the cert-cache v3 issues that has recently been reverted; keeping here as a data point, but marking as incomplete until we know a bit more.

Changed in vault-charm:
status: New → Incomplete
summary: - hook failed: "certificates-relation-changed" / AttributeError:
- 'NoneType' object has no attribute 'encode'
+ (cert cache v3 issues?) hook failed: "certificates-relation-changed" /
+ AttributeError: 'NoneType' object has no attribute 'encode'
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.