instance-state: unknown

Bug #1020834 reported by René Schultz Madsen
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
maas (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

After deploy of a maas setup, the different maas managed servers report: "instance-state: unknown"

Why is that?

It's possible to juju ssh, ping the machine, ping the maas server from the instances, do dns lookup of the maas server from the instances. It's possible to do dns lookup the different instances from the maas server.

The maas server is running:

Linux orc 2.6.38-15-server #60-Ubuntu SMP Tue May 22 11:45:41 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Description: Ubuntu 12.04 LTS
Release: 12.04

The bootstrap instance is running:

Linux bootstrap 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Description: Ubuntu 12.04 LTS
Release: 12.04

If nothing is wrong, then I would suggest that it tells me eg. "instances-state: running" or something like that...

juju -v status gives me:

2012-07-04 08:33:19,328 DEBUG Initializing juju status runtime
2012-07-04 08:33:19,339 INFO Connecting to environment...
2012-07-04 08:33:19,506 DEBUG Connecting to environment using bootstrap.mydomain.tld...
2012-07-04 08:33:19,507 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="bootstrap..mydomain.tld" remote_port="2181" local_port="60337".
Enter passphrase for key '/root/.ssh/id_rsa':
2012-07-04 08:33:22,534:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5
2012-07-04 08:33:22,534:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@662: Client environment:host.name=orc
2012-07-04 08:33:22,534:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-07-04 08:33:22,535:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@670: Client environment:os.arch=2.6.38-15-server
2012-07-04 08:33:22,535:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@671: Client environment:os.version=#60-Ubuntu SMP Tue May 22 11:45:41 UTC 2012
2012-07-04 08:33:22,536:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@679: Client environment:user.name=root
2012-07-04 08:33:22,536:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@687: Client environment:user.home=/root
2012-07-04 08:33:22,536:19667(0x7f2ab4e1e700):ZOO_INFO@log_env@699: Client environment:user.dir=/root
2012-07-04 08:33:22,536:19667(0x7f2ab4e1e700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:60337 sessionTimeout=10000 watcher=0x7f2ab2dbe6b0 sessionId=0 sessionPasswd=<null> context=0x349ed80 flags=0
2012-07-04 08:33:22,537:19667(0x7f2aab9d0700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:60337]
2012-07-04 08:33:22,704:19667(0x7f2aab9d0700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:60337], sessionId=0x138333212e8006d, negotiated timeout=10000
2012-07-04 08:33:22,706 DEBUG Environment is initialized.
2012-07-04 08:33:22,707 INFO Connected to environment.
2012-07-04 08:33:22,828:19667(0x7f2aab9d0700):ZOO_ERROR@handle_socket_error_msg@1603: Socket [127.0.0.1:60337] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
machines:
  0:
    agent-state: running
    dns-name: bootstrap.mydomain.tld
    instance-id: /MAAS/api/1.0/nodes/node-8513e6e8-c116-11e1-a8c8-00163e8a2fa6/
    instance-state: unknown
services: {}
2012-07-04 08:33:22,831 INFO 'status' command finished successfully

Tags: juju
tags: added: juju
Revision history for this message
James Page (james-page) wrote :

Hi Rene

Looking at some of the MAAS deployment I use the 'instance-state: unknown' status is normal so is a bit of a red herring in this case.

It looks like the juju zookeeper integration can't get data from zookeeper; so this is probably a juju issue rather than maas.

Please could you SSH to the bootstrap node and dig around in /var/log/zookeeper - hopefully you might find some more information from the other end of the connection that might help diagnose this issue.

Please attach any information to this bug report.

Marking 'Incomplete' pending your response; please set back to 'New' once you have provided the requested information.

Thanks

Changed in maas (Ubuntu):
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for maas (Ubuntu) because there has been no activity for 60 days.]

Changed in maas (Ubuntu):
status: Incomplete → Expired
Revision history for this message
wickwire (wickwired) wrote :

Hi,

I'm also having this issue. I have 1 MAAS Server and 1 MAAS Node.

- I have the node successfully deployed as shown in MAAS
- from the Server, I can ssh to the node without passwords - so the keys are working
- I can also ssh using juju ssh 0 just fine - I'm in
- inside the node, I have internet access - I can manually install packages
- however, I cannot get a reaction when deploying juju charms from the Server...
- I have disabled the firewall on the server (ufw disable) - didn't solve

wickwire@cloudhost:~$ juju -v status
2012-09-28 18:11:06,997 DEBUG Initializing juju status runtime
2012-09-28 18:11:07,002 INFO Connecting to environment...
2012-09-28 18:11:07,202 DEBUG Connecting to environment using node-1cc1deb45c31.local...
2012-09-28 18:11:07,202 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node-1cc1deb45c31.local" remote_port="2181" local_port="38048".
2012-09-28 18:11:07,712:18130(0x7f88a1190700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5
2012-09-28 18:11:07,712:18130(0x7f88a1190700):ZOO_INFO@log_env@662: Client environment:host.name=cloudhost
2012-09-28 18:11:07,712:18130(0x7f88a1190700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-09-28 18:11:07,712:18130(0x7f88a1190700):ZOO_INFO@log_env@670: Client environment:os.arch=3.2.0-29-generic
2012-09-28 18:11:07,712:18130(0x7f88a1190700):ZOO_INFO@log_env@671: Client environment:os.version=#46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012
2012-09-28 18:11:07,713:18130(0x7f88a1190700):ZOO_INFO@log_env@679: Client environment:user.name=wickwire
2012-09-28 18:11:07,713:18130(0x7f88a1190700):ZOO_INFO@log_env@687: Client environment:user.home=/home/wickwire
2012-09-28 18:11:07,713:18130(0x7f88a1190700):ZOO_INFO@log_env@699: Client environment:user.dir=/home/wickwire
2012-09-28 18:11:07,713:18130(0x7f88a1190700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:38048 sessionTimeout=10000 watcher=0x7f889f1346b0 sessionId=0 sessionPasswd=<null> context=0x29f9d30 flags=0
2012-09-28 18:11:07,713:18130(0x7f8897df3700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:38048]
2012-09-28 18:11:07,999:18130(0x7f8897df3700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:38048], sessionId=0x13a0c413f830010, negotiated timeout=10000
2012-09-28 18:11:08,001 DEBUG Environment is initialized.
2012-09-28 18:11:08,002 INFO Connected to environment.
2012-09-28 18:11:08,128:18130(0x7f8897df3700):ZOO_ERROR@handle_socket_error_msg@1603: Socket [127.0.0.1:38048] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
machines:
  0:
    agent-state: running
    dns-name: node-1cc1deb45c31.local
    instance-id: /MAAS/api/1.0/nodes/node-08eea08c-0944-11e2-8282-00241dfe1a0e/
    instance-state: unknown
services: {}
2012-09-28 18:11:08,130 INFO 'status' command finished successfully
wickwire@cloudhost:~$

I will include more output if needed

Changed in maas (Ubuntu):
status: Expired → Confirmed
Revision history for this message
wickwire (wickwired) wrote :

Follow up on this

- juju destroy-environment
- juju bootstrap

won't solve it

again tried to deploy a mysql charm:

wickwire@cloudhost:~$ juju deploy mysql
2012-09-28 18:13:14,531 INFO Searching for charm cs:precise/mysql in charm store
2012-09-28 18:13:15,057 INFO Connecting to environment...
2012-09-28 18:13:16,112 INFO Connected to environment.
2012-09-28 18:13:16,135 INFO Using cached charm version of mysql
2012-09-28 18:13:16,343 INFO Charm deployed as service: 'mysql'
2012-09-28 18:13:16,344 INFO 'deploy' command finished successfully
wickwire@cloudhost:~$ juju -v status
2012-09-28 18:13:21,862 DEBUG Initializing juju status runtime
2012-09-28 18:13:21,868 INFO Connecting to environment...
2012-09-28 18:13:22,060 DEBUG Connecting to environment using node-1cc1deb45c31.local...
2012-09-28 18:13:22,060 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node-1cc1deb45c31.local" remote_port="2181" local_port="57995".
2012-09-28 18:13:22,569:18171(0x7f0e907ff700):ZOO_INFO@log_env@658: Client environment:zookeeper.version=zookeeper C client 3.3.5
2012-09-28 18:13:22,569:18171(0x7f0e907ff700):ZOO_INFO@log_env@662: Client environment:host.name=cloudhost
2012-09-28 18:13:22,569:18171(0x7f0e907ff700):ZOO_INFO@log_env@669: Client environment:os.name=Linux
2012-09-28 18:13:22,569:18171(0x7f0e907ff700):ZOO_INFO@log_env@670: Client environment:os.arch=3.2.0-29-generic
2012-09-28 18:13:22,569:18171(0x7f0e907ff700):ZOO_INFO@log_env@671: Client environment:os.version=#46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012
2012-09-28 18:13:22,570:18171(0x7f0e907ff700):ZOO_INFO@log_env@679: Client environment:user.name=wickwire
2012-09-28 18:13:22,570:18171(0x7f0e907ff700):ZOO_INFO@log_env@687: Client environment:user.home=/home/wickwire
2012-09-28 18:13:22,570:18171(0x7f0e907ff700):ZOO_INFO@log_env@699: Client environment:user.dir=/home/wickwire
2012-09-28 18:13:22,570:18171(0x7f0e907ff700):ZOO_INFO@zookeeper_init@727: Initiating client connection, host=localhost:57995 sessionTimeout=10000 watcher=0x7f0e8e7a36b0 sessionId=0 sessionPasswd=<null> context=0x19677c0 flags=0
2012-09-28 18:13:22,571:18171(0x7f0e8b4fd700):ZOO_INFO@check_events@1585: initiated connection to server [127.0.0.1:57995]
2012-09-28 18:13:22,594:18171(0x7f0e8b4fd700):ZOO_INFO@check_events@1632: session establishment complete on server [127.0.0.1:57995], sessionId=0x13a0c413f830012, negotiated timeout=10000
2012-09-28 18:13:22,596 DEBUG Environment is initialized.
2012-09-28 18:13:22,597 INFO Connected to environment.
2012-09-28 18:13:22,766:18171(0x7f0e8b4fd700):ZOO_ERROR@handle_socket_error_msg@1603: Socket [127.0.0.1:57995] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
machines:
  0:
    agent-state: running
    dns-name: node-1cc1deb45c31.local
    instance-id: /MAAS/api/1.0/nodes/node-08eea08c-0944-11e2-8282-00241dfe1a0e/
    instance-state: unknown
  15:
    instance-id: pending
services:
  mysql:
    charm: cs:precise/mysql-8
    relations: {}
    units:
      mysql/4:
        agent-state: pending
        machine: 15
        public-address: null
2012-09-28 18:13:22,769 INFO 'status' command finished successfully
wickwire@cloudhost:~$

Revision history for this message
wickwire (wickwired) wrote :

wickwire@cloudhost:~$ juju ssh 0
2012-09-28 18:16:51,603 INFO Connecting to environment...
2012-09-28 18:16:52,337 INFO Connected to environment.
2012-09-28 18:16:52,450 INFO Connecting to machine 0 at node-1cc1deb45c31.local
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-31-generic-pae i686)

 * Documentation: https://help.ubuntu.com/

  System information as of Fri Sep 28 13:16:51 EDT 2012

  System load: 0.0 Processes: 100
  Usage of /: 0.3% of 450.77GB Users logged in: 0
  Memory usage: 5% IP address for eth0: 192.168.0.139
  Swap usage: 0%

  Graph this data and manage this system at https://landscape.canonical.com/

Last login: Fri Sep 28 13:16:16 2012 from cloudhost.local
ubuntu@node-1cc1deb45c31:~$ netstat -an | grep 2181
tcp 0 0 127.0.0.1:45239 127.0.0.1:2181 ESTABLISHED
tcp 0 0 127.0.0.1:47088 127.0.0.1:2181 TIME_WAIT
tcp 0 0 127.0.0.1:47081 127.0.0.1:2181 TIME_WAIT
tcp 0 0 127.0.0.1:47082 127.0.0.1:2181 TIME_WAIT
tcp 0 0 127.0.0.1:45238 127.0.0.1:2181 ESTABLISHED
tcp 0 0 127.0.0.1:47087 127.0.0.1:2181 TIME_WAIT
tcp6 0 0 :::2181 :::* LISTEN
tcp6 0 0 127.0.0.1:2181 127.0.0.1:45238 ESTABLISHED
tcp6 0 0 127.0.0.1:2181 127.0.0.1:45239 ESTABLISHED
ubuntu@node-1cc1deb45c31:~$

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.