Bug #587340 “otherwise live instance goes deaf connection refuse...” : Bugs : eucalyptus package : Ubuntu

Revision history for this message

Walt Corey (waltc) wrote on 2010-05-29:

#1

Download full text (6.8 KiB)

There is more history and documentation in Question #112157

Effectively I am running the cluster controller of a managednvlan UEC on an instance of Ubuntu 10.4 Desktop. What I had noticed, over time is running instances would go deaf, often times with what looks like valid IP addresses listed in the describe-instances output. Both the public and private IP addresses would be unavailable.
What's odd is that in the case of an intervening vpn session after the vpn session was closed, the ip endpoints to that cloud instance were removed.

Even if there were a separate dedicated cc would not one lose connectivity from the client machine? My existing (and previously existing) network was the 192.168.0.xxx served by my wireless router. The segment reserved for the cloud instances public IP address was 192.168.3.0->3.50 or some such limited range.

What is avahi and why is it withdrawing the endpoint IPs to the cloud instance?

More information...consider this:
May 28 18:26:19 cor720 NetworkManager: <info> Maximum Segment Size (MSS): 0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 10.0.0.0/8 Next Hop: 10.0.0.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.251.0/24 Next Hop: 192.168.251.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.22.0/24 Next Hop: 192.168.22.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.23.0/24 Next Hop: 192.168.23.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.24.0/24 Next Hop: 192.168.24.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 63.131.134.0/24 Next Hop: 63.131.134.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 208.111.81.157/32 Next Hop: 208.111.81.157
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 208.111.81.159/32 Next Hop: 208.111.81.159
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 72.20.25.16/32 Next Hop: 72.20.25.16
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 209.249.222.54/32 Next Hop: 209.249.222.54
May 28 18:26:19 cor720 NetworkManager: <info> Internal IP4 DNS: 10.50.33.21
May 28 18:26:19 cor720 NetworkManager: <info> Internal IP4 DNS: 10.5.4.1
May 28 18:26:19 cor720 NetworkManager: <info> DNS Domain: 'na.global.ad'
May 28 18:26:19 cor720 NetworkManager: <info> Login Banner:
May 28 18:26:19 cor720 NetworkManager: <info> -----------------------------------------
May 28 18:26:19 cor720 NetworkManager: <info> -----------------------------------------
May 28 18:26:19 cor720 vpnc[3537]: can't open pidfile /var/run/vpnc/pid for writing
May 28 18:26:20 cor720 NetworkManager: <info> VPN connection 'Monster (Maynard)' (IP Config Get) complete.
May 28 18:26:20 cor720 NetworkManager: <info> Policy set 'Monster (Maynard)' (tun0) as default for routing and DNS.
May 28 18:26:20 cor720 vmnetBridge: RTM_NEWROUTE: index:5
May 28 18:26:20 cor720 NetworkManager: <info> VPN plugin state changed: 4
May 28 18:26:20 cor720 nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/01ifupdown' exited with error status 1.
May 28 18:27:36 cor720 dhcpd: DHCPREQUEST for 172.19.1.2 from d0:0d:30:cf:06:f7 via eth0
May 28 18:27:36 cor720 dhcpd: DHCPACK on 172.19.1...

There is more history and documentation in Question #112157

Effectively I am running the cluster controller of a managednvlan UEC on an instance of Ubuntu 10.4 Desktop. What I had noticed, over time is running instances would go deaf, often times with what looks like valid IP addresses listed in the describe-instances output. Both the public and private IP addresses would be unavailable. 
What's odd is that in the case of an intervening vpn session after the vpn session was closed, the ip endpoints to that cloud instance were removed.

Even if there were a separate dedicated cc would not one lose connectivity from the client machine? My existing (and previously existing) network was the 192.168.0.xxx served by my wireless router. The segment reserved for the cloud instances public IP address was 192.168.3.0->3.50 or some such limited range.

What is avahi and why is it withdrawing the endpoint IPs to the cloud instance?

More information...consider this:
May 28 18:26:19 cor720 NetworkManager: <info> Maximum Segment Size (MSS): 0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 10.0.0.0/8 Next Hop: 10.0.0.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.251.0/24 Next Hop: 192.168.251.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.22.0/24 Next Hop: 192.168.22.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.23.0/24 Next Hop: 192.168.23.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 192.168.24.0/24 Next Hop: 192.168.24.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 63.131.134.0/24 Next Hop: 63.131.134.0
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 208.111.81.157/32 Next Hop: 208.111.81.157
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 208.111.81.159/32 Next Hop: 208.111.81.159
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 72.20.25.16/32 Next Hop: 72.20.25.16
May 28 18:26:19 cor720 NetworkManager: <info> Static Route: 209.249.222.54/32 Next Hop: 209.249.222.54
May 28 18:26:19 cor720 NetworkManager: <info> Internal IP4 DNS: 10.50.33.21
May 28 18:26:19 cor720 NetworkManager: <info> Internal IP4 DNS: 10.5.4.1
May 28 18:26:19 cor720 NetworkManager: <info> DNS Domain: 'na.global.ad'
May 28 18:26:19 cor720 NetworkManager: <info> Login Banner:
May 28 18:26:19 cor720 NetworkManager: <info> -----------------------------------------
May 28 18:26:19 cor720 NetworkManager: <info> -----------------------------------------
May 28 18:26:19 cor720 vpnc[3537]: can't open pidfile /var/run/vpnc/pid for writing
May 28 18:26:20 cor720 NetworkManager: <info> VPN connection 'Monster (Maynard)' (IP Config Get) complete.
May 28 18:26:20 cor720 NetworkManager: <info> Policy set 'Monster (Maynard)' (tun0) as default for routing and DNS.
May 28 18:26:20 cor720 vmnetBridge: RTM_NEWROUTE: index:5
May 28 18:26:20 cor720 NetworkManager: <info> VPN plugin state changed: 4
May 28 18:26:20 cor720 nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/01ifupdown' exited with error status 1.
May 28 18:27:36 cor720 dhcpd: DHCPREQUEST for 172.19.1.2 from d0:0d:30:cf:06:f7 via eth0
May 28 18:27:36 cor720 dhcpd: DHCPACK on 172.19.1.2 to d0:0d:30:cf:06:f7 via eth0
May 28 18:28:26 cor720 vpnc[3537]: select: Interrupted system call
May 28 18:28:26 cor720 vpnc[3537]: terminated by signal: 15
May 28 18:28:26 cor720 avahi-daemon[1175]: Withdrawing address record for 172.19.1.1 on eth0.
May 28 18:28:26 cor720 avahi-daemon[1175]: Withdrawing address record for 192.168.3.100 on eth0.
May 28 18:28:27 cor720 NetworkManager: <info> Policy set 'Auto eth0' (eth0) as default for routing and DNS.
May 28 18:28:27 cor720 vmnetBridge: RTM_NEWROUTE: index:2

There was a, likely, corresponding loss of signal to the open connection to the instance:
When I say it I tried to log on again.
I tried restarting eucalyptus on the cluster as well as the eucalyptus-cc
ubuntu@ip-172-19-1-2:~$ Write failed: Broken pipe
walt@cor720:~$ ssh -i .euca/mykey.priv ubuntu@192.168.3.100
ssh: connect to host 192.168.3.100 port 22: Connection timed out
walt@cor720:~$ ssh -i .euca/mykey.priv ubuntu@192.168.3.100
ssh: connect to host 192.168.3.100 port 22: Connection timed out
walt@cor720:~$

I looked at the tail end of the console log and found this:
I also tried rebooting the instance:

Begin: Running /scripts/local-bottom ...
Done.
Done.
Begin: Running /scripts/init-bottom ...
Done.
cloud-init running: Sat, 29 May 2010 01:29:42 +0000. up 9.06 seconds

waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
  01:29:44 [ 1/100]: url error [timed out]
  01:29:47 [ 2/100]: url error [timed out]
  01:29:50 [ 3/100]: url error [timed out]
  01:29:51 [ 4/100]: url error [[Errno 113] No route to host]
  01:29:54 [ 5/100]: url error [timed out]
  01:29:57 [ 6/100]: url error [timed out]
  01:30:01 [ 7/100]: url error [timed out]
  01:30:05 [ 8/100]: url error [timed out]
  01:30:09 [ 9/100]: url error [timed out]
  01:30:13 [10/100]: url error [timed out]
  01:30:17 [11/100]: url error [timed out]
  01:30:22 [12/100]: url error [timed out]
  01:30:27 [13/100]: url error [timed out]
  01:30:32 [14/100]: url error [timed out]
  01:30:37 [15/100]: url error [timed out]
  01:30:42 [16/100]: url error [timed out]
  01:30:48 [17/100]: url error [timed out]
  01:30:54 [18/100]: url error [timed out]
  01:31:00 [19/100]: url error [[Errno 113] No route to host]
  01:31:06 [20/100]: url error [timed out]
  01:31:12 [21/100]: url error [timed out]
  01:31:19 [22/100]: url error [timed out]

The system log on the cc shows:
May 28 21:23:47 cor720 dhcpd: DHCPREQUEST for 172.19.1.2 from d0:0d:30:cf:06:f7 via eth0
May 28 21:23:47 cor720 dhcpd: DHCPACK on 172.19.1.2 to d0:0d:30:cf:06:f7 via eth0
May 28 21:23:47 cor720 dhcpd: DHCPDISCOVER from d0:0d:30:cf:06:f7 via eth0
May 28 21:23:47 cor720 dhcpd: DHCPOFFER on 172.19.1.2 to d0:0d:30:cf:06:f7 via eth0
May 28 21:23:47 cor720 dhcpd: DHCPREQUEST for 172.19.1.2 (169.254.169.254) from d0:0d:30:cf:06:f7 via eth0
May 28 21:23:47 cor720 dhcpd: DHCPACK on 172.19.1.2 to d0:0d:30:cf:06:f7 via eth0
May 28 21:24:16 cor720 dhcpd: DHCPREQUEST for 172.19.1.2 from d0:0d:30:cf:06:f7 via eth0
May 28 21:24:16 cor720 dhcpd: DHCPACK on 172.19.1.2 to d0:0d:30:cf:06:f7 via eth0
May 28 21:27:47 cor720 init: uec-component-listener main process (3949) killed by TERM signal
May 28 21:28:47 cor720 init: uec-component-listener main process (22579) killed by TERM signal
May 28 21:29:45 cor720 dhcpd: DHCPREQUEST for 172.19.1.2 from d0:0d:30:cf:06:f7 via eth0
May 28 21:29:45 cor720 dhcpd: DHCPACK on 172.19.1.2 to d0:0d:30:cf:06:f7 via eth0

The instance, while continuing to show a running state, never shows a reestablished IP address pair
RESERVATION r-4481087B admin default
INSTANCE i-30CF06F7 emi-DEF41072 0.0.0.0 0.0.0.0 running mykey 0 m1.large 1970-01-01T00:00:00.65Z cluster1 eki-F52010F2 eri-0960114A

Revision history for this message

Thierry Carrez (ttx) wrote on 2010-06-02:

#2

Which version of the eucalyptus packages are you running ?
You mention running the CC on a regular 10.04 desktop, could you give us details of your networking setup ?

Changed in eucalyptus (Ubuntu):
importance:	Undecided → Medium
status:	New → Incomplete

Revision history for this message

Walt Corey (waltc) wrote on 2010-06-02:

#3

Download full text (4.2 KiB)

I believe it is 1.6.2

I followed https://help.ubuntu.com/community/UEC/PackageInstall to a tee.
The only variation was rather than installing on a 10.4 image of Ubuntu Server I installed it on Ubuntu Desktop at 10.4.

In order to insure the public addresses given the install script of the cluster components did not interfere with anything previously in existence I gave the ip address range 192.168.3.100-192.168.3.130 or maybe it was 150, I think it was 130 though, so there would be 30 available public IPs.

I do have a VPNC configured but I do not know what magic it does with IP addresses.

My Netgear router is on 192.168.0.2->192.168.0.254, the netmask is 255.255.0.0

the output from ifconfig is:

eth0 Link encap:Ethernet HWaddr 00:1e:4f:b4:e2:b5
          inet addr:192.168.0.6 Bcast:192.168.255.255 Mask:255.255.0.0
          inet6 addr: fe80::21e:4fff:feb4:e2b5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3928276 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4744355 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2479644119 (2.4 GB) TX bytes:3283396517 (3.2 GB)
          Interrupt:17

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:23077137 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23077137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16565722660 (16.5 GB) TX bytes:16565722660 (16.5 GB)

vmnet1 Link encap:Ethernet HWaddr 00:50:56:c0:00:01
          inet addr:192.168.5.1 Bcast:192.168.5.255 Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12230 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vmnet8 Link encap:Ethernet HWaddr 00:50:56:c0:00:08
          inet addr:172.16.207.1 Bcast:172.16.207.255 Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12239 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vmnet1 and 8 is, apparently, from VMWare Workstation which I use for a Windows XP image.

According to the instructions for the aforementioned installation from packages, the range I gave UEC was a unused, isolated range.

The UEC dhcp.conf is:
walt@cor720:~$ cat /var/run/eucalyptus/net/euca-dhcp.conf
# automatically generated config file for DHCP server
default-lease-time 1200;
max-lease-time 1200;
ddns-update-style none;

shared-network euca {
subnet 172.19.1.0 netmask 255.255.255.224 {
  option subnet-mask 255.255.255.224;
  option broadcast-address 172.19.1.31;
  option domain-name-servers 19...

I believe it is 1.6.2

I followed https://help.ubuntu.com/community/UEC/PackageInstall to a tee.
The only variation was rather than installing on a 10.4 image of Ubuntu Server I installed it on Ubuntu Desktop at 10.4.

In order to insure the public addresses given the install script of the cluster components did not interfere with anything previously in existence I gave the ip address range 192.168.3.100-192.168.3.130 or maybe it was 150, I think it was 130 though, so there would be 30 available public IPs.

I do have a VPNC configured but I do not know what magic it does with IP addresses.

My Netgear router is on 192.168.0.2->192.168.0.254, the netmask is 255.255.0.0

the output from ifconfig is:

eth0      Link encap:Ethernet  HWaddr 00:1e:4f:b4:e2:b5  
          inet addr:192.168.0.6  Bcast:192.168.255.255  Mask:255.255.0.0
          inet6 addr: fe80::21e:4fff:feb4:e2b5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3928276 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4744355 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2479644119 (2.4 GB)  TX bytes:3283396517 (3.2 GB)
          Interrupt:17

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:23077137 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23077137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:16565722660 (16.5 GB)  TX bytes:16565722660 (16.5 GB)

vmnet1    Link encap:Ethernet  HWaddr 00:50:56:c0:00:01  
          inet addr:192.168.5.1  Bcast:192.168.5.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12230 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vmnet8    Link encap:Ethernet  HWaddr 00:50:56:c0:00:08  
          inet addr:172.16.207.1  Bcast:172.16.207.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12239 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vmnet1 and 8 is, apparently, from VMWare Workstation which I use for a Windows XP image.

According to the instructions for the aforementioned installation from packages, the range I gave UEC was a unused, isolated range.

The UEC dhcp.conf is:
walt@cor720:~$ cat /var/run/eucalyptus/net/euca-dhcp.conf 
# automatically generated config file for DHCP server
default-lease-time 1200;
max-lease-time 1200;
ddns-update-style none;

shared-network euca {
subnet 172.19.1.0 netmask 255.255.255.224 {
  option subnet-mask 255.255.255.224;
  option broadcast-address 172.19.1.31;
  option domain-name-servers 192.168.0.1, 127.0.0.1;
  option routers 172.19.1.1;
}

host node-172.19.1.2 {
  hardware ethernet d0:0d:41:31:06:66;
  fixed-address 172.19.1.2;
}

host node-172.19.1.3 {
  hardware ethernet d0:0d:3E:C4:07:F8;
  fixed-address 172.19.1.3;
}

host node-172.19.1.4 {
  hardware ethernet d0:0d:39:38:07:41;
  fixed-address 172.19.1.4;
}

host node-172.19.1.5 {
  hardware ethernet d0:0d:58:A4:0A:49;
  fixed-address 172.19.1.5;
}
}

The single node controller machine was installed from a Ubuntu Server 10.4 cd and is reserved the address 192.168.0.12 on the aforementioned wireless router. On it I constructed the bridged network precisely to the instructions given in the url noted above:

interface=eth0
bridge=br0
sudo sed -i "s/^iface $interface inet $.*$$/iface $interface inet manual\n\nauto br0\niface $bridge inet \1/" /etc/network/interfaces
sudo tee -a /etc/network/interfaces <<EOF
        bridge_ports $interface
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off
EOF

Feel free to request whatever else I can provide to assist you.
Thanks,
Walt

Revision history for this message

Walt Corey (waltc) wrote on 2010-06-08:

#4

OK, you asked for more information. I gave you the requested information. Could this bug be placed in a status where it won't self destruct if nobody picks it up in 53 days?

Thanks

Revision history for this message

Walt Corey (waltc) wrote on 2010-06-09:

#5

What I discovered, which means there are several paths to this problem, is the private address, in this case, remained valid. I can connect via the private address.

I tried to connect then release with the intent of connect the original ip address of 192.168.3.100 but in both cases got the error "Address: Permission denied while trying to release address: 192.168.3.100"

The same error occurred when prefixed by sudo.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2010-12-07:

#6

[Expired for eucalyptus (Ubuntu) because there has been no activity for 60 days.]

Changed in eucalyptus (Ubuntu):
status:	Incomplete → Expired

Ubuntu
eucalyptus package

otherwise live instance goes deaf connection refused

Bug Description

Other bug subscribers

Related questions

Remote bug watches

Ubuntueucalyptus package

otherwise live instance goes deaf connection refused

Bug Description

Other bug subscribers

Related questions

Remote bug watches

Ubuntu
eucalyptus package