unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 #007Caution: invalid backup GPT header, but valid main header; regenerating
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 backup header from main header.
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 on the recovery & transformation menu to examine the two tables.
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 Warning! One or more CRCs don't match. You should repair the disk!
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 #015Building dependency tree... 0%#015#015Building dependency tree... 0%#015#015Building dependency tree... 50%#015#015Building dependency tree... 50%#015#015Building dependency tree
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 #015Reading state information... 0%#015#015Reading state information... 0%#015#015Reading state information... Done
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 2768, in <module>
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 main()
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 2746, in main
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 args.func(args)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 1390, in main_prepare
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 zap(args.data)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 1013, in zap
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 with file(dev, 'wb') as dev_file:
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 IOError: [Errno 123] No medium found: '/dev/sdb'
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR unit.ceph/1.juju-log cmd.go:247 mon:3: Unable to initialize device: /dev/sdb
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/mon-relation-changed", line 312, in <module>
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 hooks.execute(sys.argv)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/charmhelpers/core/hookenv.py", line 508, in execute
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 self._hooks[hook_name]()
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/mon-relation-changed", line 196, in mon_relation
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 reformat_osd(), config('ignore-device-errors'))
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/ceph.py", line 327, in osdize
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 osdize_dev(dev, osd_format, osd_journal, reformat_osd, ignore_errors)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/ceph.py", line 375, in osdize_dev
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 raise e
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 subprocess.CalledProcessError: Command '['ceph-disk-prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR juju.worker.uniter.operation runhook.go:86 hook "mon-relation-changed" failed: exit status 1
This looks like the operative failure. I'd suggest looking into possible hardware failures on that disk. I'll add ceph on here so perhaps the maintainers of that project could have a chance to sound in.
http:// paste.ubuntu. com/10991033/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/ 2.mon-relation- changed logger.go:40 #007Caution: invalid backup GPT header, but valid main header; regenerating 2.mon-relation- changed logger.go:40 backup header from main header. 2.mon-relation- changed logger.go:40 2.mon-relation- changed logger.go:40 Warning! Main and backup partition tables differ! Use the 'c' and 'e' options 2.mon-relation- changed logger.go:40 on the recovery & transformation menu to examine the two tables. 2.mon-relation- changed logger.go:40 2.mon-relation- changed logger.go:40 Warning! One or more CRCs don't match. You should repair the disk! 2.mon-relation- changed logger.go:40 1.mon-relation- changed logger.go:40 #015Building dependency tree... 0%#015#015Building dependency tree... 0%#015#015Building dependency tree... 50%#015#015Building dependency tree... 50%#015#015Building dependency tree 1.mon-relation- changed logger.go:40 #015Reading state information... 0%#015#015Reading state information... 0%#015#015Reading state information... Done 1.mon-relation- changed logger.go:40 Traceback (most recent call last): 1.mon-relation- changed logger.go:40 File "/usr/sbin/ ceph-disk" , line 2768, in <module> 1.mon-relation- changed logger.go:40 main() 1.mon-relation- changed logger.go:40 File "/usr/sbin/ ceph-disk" , line 2746, in main 1.mon-relation- changed logger.go:40 args.func(args) 1.mon-relation- changed logger.go:40 File "/usr/sbin/ ceph-disk" , line 1390, in main_prepare 1.mon-relation- changed logger.go:40 zap(args.data) 1.mon-relation- changed logger.go:40 File "/usr/sbin/ ceph-disk" , line 1013, in zap 1.mon-relation- changed logger.go:40 with file(dev, 'wb') as dev_file: 1.mon-relation- changed logger.go:40 IOError: [Errno 123] No medium found: '/dev/sdb' 1.juju- log cmd.go:247 mon:3: Unable to initialize device: /dev/sdb 1.mon-relation- changed logger.go:40 Traceback (most recent call last): 1.mon-relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- 1/charm/ hooks/mon- relation- changed" , line 312, in <module> 1.mon-relation- changed logger.go:40 hooks.execute( sys.argv) 1.mon-relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- 1/charm/ hooks/charmhelp ers/core/ hookenv. py", line 508, in execute 1.mon-relation- changed logger.go:40 self._hooks[ hook_name] () 1.mon-relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- 1/charm/ hooks/mon- relation- changed" , line 196, in mon_relation 1.mon-relation- changed logger.go:40 reformat_osd(), config( 'ignore- device- errors' )) 1.mon-relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- 1/charm/ hooks/ceph. py", line 327, in osdize 1.mon-relation- changed logger.go:40 osdize_dev(dev, osd_format, osd_journal, reformat_osd, ignore_errors) 1.mon-relation- changed logger.go:40 File "/var/lib/ juju/agents/ unit-ceph- 1/charm/ hooks/ceph. py", line 375, in osdize_dev 1.mon-relation- changed logger.go:40 raise e 1.mon-relation- changed logger.go:40 subprocess. CalledProcessEr ror: Command '['ceph- disk-prepare' , '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1 uniter. operation runhook.go:86 hook "mon-relation- changed" failed: exit status 1
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR juju.worker.
This looks like the operative failure. I'd suggest looking into possible hardware failures on that disk. I'll add ceph on here so perhaps the maintainers of that project could have a chance to sound in.