[19.04] keystone leader unit sometimes fails to add endpoints when "certificates" relation is present but certs not installed and TLS not configured.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Keystone Charm |
Triaged
|
High
|
Unassigned |
Bug Description
1) deployed an HA bundle without TLS enabled;
2) related all openstack services to vault via "certificates" relation;
3) observed keystone trying to use http://
LISTEN 0 128 *:35347 *:* users:(
LISTEN 0 128 *:35357 *:* users:(
https:/
unit-keystone-0: 18:23:09 DEBUG unit.keystone/
# ...
unit-keystone-0: 18:23:09 ERROR juju.worker.
This keystone unit tried to add new (https) endpoints of other openstack services into the catalog using the following code (clustered & https => public_port - 20 = 35337):
https:/
However, /etc/apache2/
Listen 35347
Listen 4990
<VirtualHost *:35347>
# ...
juju show-status-log keystone/0 --days 1
Time Type Status Message
03 May 2019 11:24:08Z workload waiting waiting for machine
03 May 2019 11:24:08Z juju-unit allocating
03 May 2019 11:42:56Z workload waiting installing agent
03 May 2019 11:42:59Z workload waiting agent initializing
03 May 2019 11:43:04Z workload maintenance installing charm software
03 May 2019 11:43:04Z juju-unit executing running install hook
03 May 2019 11:43:56Z workload maintenance Executing pre-install
03 May 2019 11:44:01Z workload maintenance Installing apt packages
03 May 2019 11:47:36Z juju-unit executing running leader-elected hook
03 May 2019 11:47:52Z juju-unit executing running config-changed hook
03 May 2019 11:49:22Z juju-unit executing running start hook
03 May 2019 11:49:32Z workload blocked Missing relations: database
03 May 2019 11:49:36Z juju-unit executing running cluster-
03 May 2019 11:53:21Z juju-unit executing running identity-
03 May 2019 11:53:36Z juju-unit executing running commands
03 May 2019 11:54:36Z juju-unit executing running identity-
03 May 2019 11:55:27Z juju-unit executing running commands
03 May 2019 11:55:46Z juju-unit executing running identity-
03 May 2019 11:56:13Z juju-unit executing running identity-
03 May 2019 11:57:43Z juju-unit executing running identity-
03 May 2019 11:58:25Z juju-unit executing running identity-
03 May 2019 11:58:53Z juju-unit executing running identity-
03 May 2019 11:59:07Z juju-unit executing running cluster-
03 May 2019 11:59:30Z juju-unit executing running identity-
03 May 2019 11:59:44Z juju-unit executing running identity-
03 May 2019 12:00:13Z juju-unit executing running commands
03 May 2019 12:00:20Z juju-unit executing running identity-
03 May 2019 12:01:03Z juju-unit executing running shared-
03 May 2019 12:01:19Z juju-unit executing running identity-
03 May 2019 12:01:35Z juju-unit executing running identity-
03 May 2019 12:01:49Z juju-unit executing running ha-relation-joined hook
03 May 2019 12:02:37Z juju-unit executing running identity-
03 May 2019 12:02:51Z juju-unit executing running ha-relation-changed hook
03 May 2019 12:03:10Z juju-unit executing running identity-
03 May 2019 12:03:26Z juju-unit executing running identity-
03 May 2019 12:03:41Z juju-unit executing running identity-
03 May 2019 12:03:57Z juju-unit executing running identity-
03 May 2019 12:04:28Z juju-unit executing running identity-
03 May 2019 12:05:13Z workload waiting Incomplete relations: database
03 May 2019 12:05:15Z juju-unit executing running commands
03 May 2019 12:05:23Z juju-unit executing running shared-
03 May 2019 12:08:11Z juju-unit executing running shared-
03 May 2019 12:08:28Z juju-unit executing running identity-
03 May 2019 12:08:43Z juju-unit executing running shared-
03 May 2019 12:11:48Z juju-unit executing running commands
03 May 2019 12:11:59Z juju-unit executing running identity-
03 May 2019 12:13:04Z juju-unit executing running identity-
03 May 2019 12:13:20Z juju-unit executing running identity-
03 May 2019 12:15:13Z juju-unit executing running commands
03 May 2019 12:15:19Z juju-unit executing running identity-
03 May 2019 12:16:44Z juju-unit executing running identity-
03 May 2019 12:17:15Z juju-unit executing running identity-
03 May 2019 12:19:06Z juju-unit executing running identity-
03 May 2019 12:19:23Z juju-unit executing running identity-
03 May 2019 12:19:46Z juju-unit idle
03 May 2019 12:20:02Z juju-unit executing running commands
03 May 2019 12:20:08Z juju-unit idle
#...
03 May 2019 17:40:02Z juju-unit executing running commands
03 May 2019 17:40:08Z juju-unit idle
03 May 2019 17:45:01Z juju-unit executing running commands
03 May 2019 17:45:08Z juju-unit idle
03 May 2019 17:45:35Z juju-unit executing running certificates-
03 May 2019 17:45:53Z juju-unit executing running certificates-
03 May 2019 17:46:11Z juju-unit executing running certificates-
03 May 2019 17:46:25Z workload active Unit is ready
03 May 2019 17:46:40Z juju-unit executing running identity-
03 May 2019 17:47:44Z juju-unit error hook failed: "identity-
03 May 2019 17:47:49Z juju-unit executing running identity-
03 May 2019 17:48:47Z juju-unit error hook failed: "identity-
03 May 2019 17:48:52Z juju-unit executing running action juju-run
03 May 2019 17:48:52Z juju-unit error hook failed: "identity-
03 May 2019 17:48:56Z juju-unit executing running identity-
03 May 2019 17:49:52Z juju-unit error hook failed: "identity-
03 May 2019 17:50:02Z juju-unit executing running commands
# keystone/0
root@juju-
certificates:106
root@juju-
vault/0
vault/1
root@juju-
ca: |-
-----BEGIN CERTIFICATE-----
#...
-----END CERTIFICATE-----
client.cert: |-
-----BEGIN CERTIFICATE-----
# ...
-----END CERTIFICATE-----
client.key: |-
-----BEGIN RSA PRIVATE KEY-----
# ...
-----END RSA PRIVATE KEY-----
egress-subnets: 10.232.7.208/32
ingress-address: 10.232.7.208
private-address: 10.232.7.208
root@juju-
egress-subnets: 10.232.7.234/32
ingress-address: 10.232.7.234
private-address: 10.232.7.234
The unit log reported "Unit is ready" after certificates-
2019-05-03 17:46:09 DEBUG certificates-
2019-05-03 17:46:09 DEBUG certificates-
2019-05-03 17:46:09 DEBUG certificates-
2019-05-03 17:46:09 INFO juju-log certificates:106: Unit is ready
root@juju-
total 8
dr-xr-xr-x 2 root root 4096 May 3 17:46 .
dr-xr-xr-x 3 root root 4096 May 3 17:46 ..
The second keystone unit has everything (certs, WSGI daemon listening on the correct port etc.).
While trying to replay what went wrong I can see that processed_requests are not exposed for keystone/0:
ipdb> certs = data.get(
ipdb> certs
It is not clear to me how this unit got into that state.
It is worth noting that the lab deployment used vault + totally-
I will leave this for reference in the 'incomplete' state because this may be quite difficult to reproduce or debug.
Changed in charm-keystone: | |
status: | New → Incomplete |
Changed in charm-keystone: | |
milestone: | none → 19.07 |
Changed in charm-keystone: | |
assignee: | nobody → Ryan Beisner (1chb1n) |
status: | Confirmed → Incomplete |
Changed in charm-keystone: | |
milestone: | 19.07 → 19.10 |
Changed in charm-keystone: | |
milestone: | 19.10 → 20.01 |
Changed in charm-keystone: | |
milestone: | 20.01 → 20.05 |
Changed in charm-keystone: | |
assignee: | James Page (james-page) → nobody |
Changed in charm-keystone: | |
milestone: | 20.05 → 20.08 |
Changed in charm-keystone: | |
assignee: | nobody → Alex Kavanagh (ajkavanagh) |
Changed in charm-keystone: | |
milestone: | 20.08 → none |
Changed in charm-keystone: | |
status: | In Progress → Triaged |
Reproduced it on a clean 19.04+Queens deployment without totally- insecure- unlock (did a manual unlocking procedure by hand):
VAULT_UNIT_ IP=$(juju run --unit vault/0 "network-get access --ingress- address= true"); export VAULT_ADDR="http:// $VAULT_ UNIT_IP: 8200"
vault operator init -key-shares=1 -key-threshold=1 > bundles/vault.txt vault-txt>
vault operator unseal <key-from-
VAULT_UNIT_ IP=$(juju run --unit vault/1 "network-get access --ingress- address= true"); export VAULT_ADDR="http:// $VAULT_ UNIT_IP: 8200"
vault operator unseal <key>
export VAULT_TOKEN= <initial- root-token- from-vault. txt>
vault token create --ttl=10m
juju export-bundle /pastebin. canonical. com/p/cCBN5PnYR x/
https:/
juju show-status-log keystone/0 --days 1 /paste. ubuntu. com/p/36fJJpXJ4 Q/
https:/
keystone/0* error idle 0/lxd/5 10.232.46.164 5000/tcp hook failed: "identity- service- relation- changed" keystone/ 1 active idle 10.232.46.164 Unit is ready and clustered saml-mellon/ 1 active idle 10.232.46.164 Unit is ready keystone/ 0* active idle 10.232.46.157 Unit is ready and clustered saml-mellon/ 0* active idle 10.232.46.157 Unit is ready
hacluster-
keystone-
keystone/1 active idle 1/lxd/5 10.232.46.157 5000/tcp Unit is ready
hacluster-
keystone-
https:/ /private- fileshare. canonical. com/~dima/ charm-dumps/ 10-05-2019- keystone- 0-var-log- etc.tar. gz
https:/ /private- fileshare. canonical. com/~dima/ charm-dumps/ 10-05-2019- var-lib- juju-agents- keystone- 0-hacluster. tar.gz
sqlite3 /var/lib/ juju/agents/ unit-keystone- 0/charm/ .unit-state. db |["0"] juju/agents/ unit-keystone- 0/charm" , "JUJU_CHARM_DIR": "/var/lib/ juju/agents/ unit-keystone- 0/charm" , "JUJU_CONTEXT_ID": "keystone/ 0-identity- service- relation- changed- 866489089749738 405", "JUJU_AGENT_ SOCKET" : "@/var/ lib/juju/ agents/ unit-keystone- 0/agent. socket" , "JUJU_UNIT_NAME": "keystone/0", "JUJU_MODEL_UUID": "bfd89e39- f481-4e7f- 8a1c-fd9f4a35b8 98", "JUJU_MODEL_NAME": "openstack", "JUJU_API_ ADDRESSES" : "10.232. 1.60:17070" , "JUJU_SLA": "unsupported", "JUJU_MACHINE_ID": "0/lxd/5", "JUJU_PRINCIPAL _UNIT": "", "JUJU_AVAILABIL ITY_ZONE" : "default", "JUJU_VERSION": "2.6-rc2", "CLOUD_ API_VERSION" : "", "JUJU_CHARM_ HTTP_PROXY" : "http:// 10.232. 0.1:3128", "JUJU_CHARM_ HTTPS_PROXY" : "http:// 10.232. 0.1:3128", "JUJU_CHARM_ FTP_PROXY" : "", "JUJU_CHARM_ NO_PROXY" : "10.0.0. 0/8,192. 168.0.0/ 16,172. 16.0.0/ 12", "JUJU_METER_ STATUS" : "AMBER", "JUJU_METER_INFO": "not set", "JUJU_RELATION": "identity-service", "JUJU_RELATION_ID": "identity- service: 38", "JUJU_REMOTE_UNIT": "cinder/1", "APT_LISTCHANGE S_FRONTEND" : "none", "DEBIAN_FRONTEND": "noninteractive", "PATH": "/var/lib/ juju/tools/ unit-keystone- 0:/usr/ local/sbin: /usr/local/ bin:/usr/ sbin:/usr/ bin:/sbin: /bin"} identity- service: 38" "<redacted> " nonce-keystone- fid-service- provider: 92|...
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> select * from kv;
charm_revisions
env|{"CHARM_DIR": "/var/lib/
unit|"keystone/0"
relid|"
stat-password|
fid-restart-