Charm Helpers

hookenv.relation_get raises CalledProcessError

Bug #1614934 reported by Stuart Bishop
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charm Helpers
Triaged
Critical
Unassigned
PostgreSQL Charm
Won't Fix
High
Unassigned

Bug Description

relation-get fails if a standby attempts to query the master's connection details too early.

2016-08-19 11:44:30 INFO db-admin-relation-changed error: cannot read settings for unit "postgresql/2" in relation "telegraf:postgresql postgresql:db-admin": settings not found
2016-08-19 11:44:30 INFO db-admin-relation-changed Traceback (most recent call last):
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/hooks/db-admin-relation-changed", line 18, in <module>
2016-08-19 11:44:30 INFO db-admin-relation-changed main()
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/__init__.py", line 78, in main
2016-08-19 11:44:30 INFO db-admin-relation-changed bus.dispatch()
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/bus.py", line 434, in dispatch
2016-08-19 11:44:30 INFO db-admin-relation-changed _invoke(other_handlers)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/bus.py", line 417, in _invoke
2016-08-19 11:44:30 INFO db-admin-relation-changed handler.invoke()
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charms/reactive/bus.py", line 291, in invoke
2016-08-19 11:44:30 INFO db-admin-relation-changed self._action(*args)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/reactive/postgresql/client.py", line 79, in mirror_master
2016-08-19 11:44:30 INFO db-admin-relation-changed db_relation_mirror(rel)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/reactive/postgresql/client.py", line 149, in db_relation_mirror
2016-08-19 11:44:30 INFO db-admin-relation-changed rel.local.update({k: master_info.get(k) for k in master_keys})
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/var/lib/juju/agents/unit-postgresql-3/charm/reactive/postgresql/client.py", line 149, in <dictcomp>
2016-08-19 11:44:30 INFO db-admin-relation-changed rel.local.update({k: master_info.get(k) for k in master_keys})
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/lib/python3.4/_collections_abc.py", line 422, in get
2016-08-19 11:44:30 INFO db-admin-relation-changed return self[key]
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/lib/python3.4/collections/__init__.py", line 904, in __getitem__
2016-08-19 11:44:30 INFO db-admin-relation-changed if key in self.data:
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charmhelpers/context.py", line 163, in data
2016-08-19 11:44:30 INFO db-admin-relation-changed return hookenv.relation_get(rid=self.relid, unit=self.unit)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charmhelpers/core/hookenv.py", line 72, in wrapper
2016-08-19 11:44:30 INFO db-admin-relation-changed res = func(*args, **kwargs)
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/local/lib/python3.4/dist-packages/charmhelpers/core/hookenv.py", line 357, in relation_get
2016-08-19 11:44:30 INFO db-admin-relation-changed return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2016-08-19 11:44:30 INFO db-admin-relation-changed File "/usr/lib/python3.4/subprocess.py", line 620, in check_output
2016-08-19 11:44:30 INFO db-admin-relation-changed raise CalledProcessError(retcode, process.args, output=output)
2016-08-19 11:44:30 INFO db-admin-relation-changed subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-r', 'db-admin:25', '-', 'postgresql/2']' returned non-zero exit status 1

Stuart Bishop (stub)
Changed in postgresql-charm:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Stuart Bishop (stub) wrote :

The bug here is that charmhelpers.core.hookenv.relation_get() may raise a subprocess.CalledProcessError when attempting to access a relation that is not fully setup. While this is arguably a Juju bug, we need to make relation_get more robust and handle the exception. Maybe log a WARNING or ERROR too.

Changed in charm-helpers:
status: New → Triaged
importance: Undecided → Critical
Changed in postgresql-charm:
status: Triaged → Won't Fix
summary: - Race condition in client connection
+ hookenv.relation_get raises CalledProcessError
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.