Juju Charms Collection

  1. Bug #1577955
  2. Activity log

Activity log for bug #1577955

Date Who What changed Old value New value Message
2016-05-03 21:51:31 Mike McCracken bug added bug
2016-05-03 21:52:22 Mike McCracken description A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/ unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC
2016-05-03 22:37:29 Mike McCracken description A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/ unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) this is on a LXD controller, on xenial, with juju 2.0 beta 6. Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/ unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC
2016-05-03 22:39:53 Mike McCracken description A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) this is on a LXD controller, on xenial, with juju 2.0 beta 6. Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/ unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) this is on a LXD controller, on xenial, with juju 2.0 beta 6. this is part of deploying this bundle: http://paste.ubuntu.com/16209402/ Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/ unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC
2016-05-04 20:31:02 Mike McCracken summary cassandra fails on 'leader-elected' hook cassandra fails on 'leader-elected' hook because it doesn't know about lxd
2016-05-04 20:31:23 Mike McCracken description A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) this is on a LXD controller, on xenial, with juju 2.0 beta 6. this is part of deploying this bundle: http://paste.ubuntu.com/16209402/ Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/ unit-cassandra-0: 2016-05-03 21:20:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:20:45.468793373 +0000 UTC unit-cassandra-0: 2016-05-03 21:20:28 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:20:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:15.46891061 +0000 UTC unit-cassandra-0: 2016-05-03 21:21:07 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["*** leader-elected Hook Start"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 *** leader-elected Hook Start unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "data"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "cluster"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Loading state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Loading state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "coordinator"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator No peer relation. Loading local state"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator No peer relation. Loading local state unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader handling coordinator requests"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader handling coordinator requests unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 coordinator.CassandraCoordinator Leader released restart from cassandra/0, held 0:16:44.795014 unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/set_proxy"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/set_proxy unit-cassandra-0: 2016-05-03 21:21:08 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/preinstall"] unit-cassandra-0: 2016-05-03 21:21:08 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/preinstall unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_meminfo"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_meminfo unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 total used free shared buffers cached unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Mem: 15G 282M 15G 17M 0B 64M unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 -/+ buffers/cache: 217M 15G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 Swap: 4.0G 143M 3.9G unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/revert_unchangeable_config"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/revert_unchangeable_config unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/store_unit_private_ip"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/store_unit_private_ip unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "private-address"] unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/add_implicit_package_signing_keys"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/add_implicit_package_signing_keys unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 OK unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_sources"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_sources unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/swapoff"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/swapoff unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 swapoff: Not superuser. unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "WARNING" "Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully"] unit-cassandra-0: 2016-05-03 21:21:09 WARNING unit.cassandra/0.juju-log server.go:269 Got an error trying to turn off swapping. Command '['swapoff', '-a']' returned non-zero exit status 1. We may be in an LXC. Exiting gracefully unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_sysctl"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_sysctl unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/sysctl.d/99-cassandra.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/sysctl.d/99-cassandra.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.leader-elected logger.go:40 sysctl: permission denied on key 'vm.max_map_count' unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_limits"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_limits unit-cassandra-0: 2016-05-03 21:21:09 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444"] unit-cassandra-0: 2016-05-03 21:21:09 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/security/limits.d/cassandra-charm.conf root:root 444 unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_oracle_jre"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_oracle_jre unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/install_cassandra_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/install_cassandra_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/install_packages"] unit-cassandra-0: 2016-05-03 21:21:10 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/install_packages unit-cassandra-0: 2016-05-03 21:21:10 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["extra_packages" "--format=json"] Reading package lists... Done21:21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:12 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:12 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/emit_java_version"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/emit_java_version unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 openjdk version "1.8.0_91" unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.leader-elected logger.go:40 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/ensure_cassandra_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/ensure_cassandra_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/ensure_package_status"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/ensure_package_status unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "config-get" ["wait_for_storage_broker" "--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:13 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_firewall"] unit-cassandra-0: 2016-05-03 21:21:13 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_firewall unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "relation-ids" ["--format=json" "database-admin"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/update_etc_hosts"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/update_etc_hosts unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Marking line '10.17.158.17 juju-27986dd3-50a8-4f33-876f-90967fb908f5-machine-0' for update or removal unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "is-leader" ["--format=json"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/maintain_seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/maintain_seeds unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Current seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Current seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Bootstrapped == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Bootstrapped == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Updated seeds == {'10.17.158.17'}"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Updated seeds == {'10.17.158.17'} unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-set" ["seeds=10.17.158.17"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:14 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "leader-get" ["--format=json" "seeds"] unit-cassandra-0: 2016-05-03 21:21:14 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Configuring seeds as '10.17.158.17'"] unit-cassandra-0: 2016-05-03 21:21:14 DEBUG unit.cassandra/0.juju-log server.go:269 Configuring seeds as '10.17.158.17' unit-cassandra-0: 2016-05-03 21:21:15 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:21:45.469018307 +0000 UTC Reading package lists... Done21:21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:16 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:16 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:16 INFO worker.uniter.jujuc server.go:173 running hook tool "unit-get" ["--format=json" "public-address"] Reading package lists... Done21:21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:18 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:18 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["* Helper leader-elected/write_cassandra_yaml"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 * Helper leader-elected/write_cassandra_yaml unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra.yaml root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra.yaml root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_env"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_env unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-env.sh root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-env.sh root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/configure_cassandra_rackdc"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/configure_cassandra_rackdc unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 Writing file /etc/cassandra/cassandra-rackdc.properties root:root 444 unit-cassandra-0: 2016-05-03 21:21:19 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["** Action leader-elected/reset_all_io_schedulers"] unit-cassandra-0: 2016-05-03 21:21:19 INFO unit.cassandra/0.juju-log server.go:269 ** Action leader-elected/reset_all_io_schedulers Reading package lists... Done21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Building dependency tree :21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Reading state information... Done1:21 INFO unit.cassandra/0.leader-elected logger.go:40 unit-cassandra-0: 2016-05-03 21:21:21 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Setting block device of /var/lib/cassandra/data to IO scheduler noop"] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.juju-log server.go:269 Setting block device of /var/lib/cassandra/data to IO scheduler noop unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 Traceback (most recent call last): unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/leader-elected", line 20, in <module> unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 hooks.default_hook() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/hooks.py", line 50, in default_hook unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 sm.manage() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 137, in manage unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.reconfigure_services() unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 191, in reconfigure_services unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 self.fire_event('data_ready', service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/charmhelpers/core/services/base.py", line 238, in fire_event unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 callback(service_name) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 104, in wrapper unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 return func(*args, **kw) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/actions.py", line 628, in reset_all_io_schedulers unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 helpers.set_io_scheduler(config['io_scheduler'], d) unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 File "/var/lib/juju/agents/unit-cassandra-0/charm/hooks/helpers.py", line 217, in set_io_scheduler unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 block_dev = re.findall(block_regex, output)[0] unit-cassandra-0: 2016-05-03 21:21:21 INFO unit.cassandra/0.leader-elected logger.go:40 IndexError: list index out of range unit-cassandra-0: 2016-05-03 21:21:21 ERROR juju.worker.uniter.operation runhook.go:107 hook "leader-elected" failed: exit status 1 unit-cassandra-0: 2016-05-03 21:21:21 INFO juju.worker.uniter resolver.go:107 awaiting error resolution for "leader-elected" hook unit-cassandra-0: 2016-05-03 21:21:45 INFO juju.worker.leadership tracker.go:182 cassandra/0 will renew cassandra leadership at 2016-05-03 21:22:15.469070115 +0000 UTC A deploy of cs:trusty/cassandra-15 is failing with the following in the debug log: (these failures repeat regularly) this is on a LXD controller, on xenial, with juju 2.0 beta 6. this is part of deploying this bundle: http://paste.ubuntu.com/16209402/ Here's a better-formatted way to see it: http://paste.ubuntu.com/16209115/