Regiond crashes: sequence item 0: expected str instance, ConnectionLost found
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Incomplete
|
Medium
|
Unassigned | ||
3.5 |
Triaged
|
Undecided
|
Unassigned |
Bug Description
Running HA MAAS using MAAS 3.5 while trying to add new bare-metal nodes regiond crashes on one of the maas hosts because the connection to the bare-metal host was closed unexpectedly.
the machine is being created in MAAS with:
maas root machines create hostname=beartic power_type=ipmi architecture=
and this is the 6th machine being added to the MAAS, the first 5 all succeeded.
from the attached logs, on host 10.244.40.32 you can see the regiond going down at the same time the call to create the machine:
Apr 9 18:56:51 swoobat regiond[1297846]: maasserver.
Apr 9 18:56:51 swoobat regiond[1297846]: Traceback (most recent call last):
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: raise exc from exc.__cause__
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: with post_commit_hooks:
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: self.fire()
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: result = func(*args, **kwargs)
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: self._fire_
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: result.
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: raise self.value.
Apr 9 18:56:51 swoobat regiond[1297846]: twisted.
Apr 9 18:56:51 swoobat regiond[1297846]: During handling of the above exception, another exception occurred:
Apr 9 18:56:51 swoobat regiond[1297846]: Traceback (most recent call last):
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: response = self.process_
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: response = middleware_
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: self.log_
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: logger.error(" Exception: %s ".center(79, "#") % str(exception))
Apr 9 18:56:51 swoobat regiond[1297846]: File "/usr/lib/
Apr 9 18:56:51 swoobat regiond[1297846]: s.append(" ".join(self.args))
Apr 9 18:56:51 swoobat regiond[1297846]: TypeError: sequence item 0: expected str instance, ConnectionLost found
Apr 9 18:56:51 swoobat regiond[1297846]: regiond: [info] 127.0.0.1 POST /MAAS/api/
I am unsure why the connection to the host was lost, but the error needs to be handled in a way that doesnt take down all of regiond, which comes up about 30 seconds later.
The logs are attached, but the testrun can be found at:
https:/
and the maas logs can also be found at:
https:/
Changed in maas: | |
milestone: | none → 3.6.0 |
status: | New → Triaged |
tags: | added: bug-council |
Just checked in the logs, the region is not actually dead after the exception
Apr 9 18:56:51 swoobat regiond[1297846]: TypeError: sequence item 0: expected str instance, ConnectionLost found 2.0/machines/ HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Python- httplib2/ 0.20.2 (gzip)) rackcontrollers : [info] Existing rack controller 'swoobat' running version 3.5.0~rc9- 16269-g. 8ccfdc1b0 has connected to region 'meinfoo'. ss(type= 'TCP', host=': :ffff:192. 168.33. 23', port=34234, flowInfo=0, scopeID=0) PEER:IPv6Addres s(type= 'TCP', host=': :ffff:192. 168.33. 22', port=5252, flowInfo=0, scopeID=0)) ss(type= 'TCP', host=': :ffff:192. 168.33. 23', port=34250, flowInfo=0, scopeID=0) PEER:IPv6Addres s(type= 'TCP', host=': :ffff:192. 168.33. 22', port=5252, flowInfo=0, scopeID=0)) ver.rpc. clusterservice: [info] Event-loop 'meinfoo: pid=2128395' authenticated. ver.rpc. clusterservice: [info] Event-loop 'meinfoo: pid=2128395' authenticated. rackcontrollers : [info] Existing rack controller 'swoobat' running version 3.5.0~rc9- 16269-g. 8ccfdc1b0 has connected to region 'meinfoo'. ver.rpc. clusterservice: [info] Rack controller '6m8qr6' registered (via meinfoo: pid=2128395) with MAAS version 3.5.0~rc9- 16269-g. 8ccfdc1b0. ver.rpc. clusterservice: [info] Rack controller '6m8qr6' registered (via meinfoo: pid=2128395) with MAAS version 3.5.0~rc9- 16269-g. 8ccfdc1b0. internet. protocol. Factory: [info] RegionServer connection established (HOST:IPv6Addre ss(type= 'TCP', host=': :ffff:10. 245.222. 223', port=5251, flowInfo=0, scopeID=0) PEER:IPv6Addres s(type= 'TCP', host=': :ffff:10. 245.222. 182', port=34344, flowInfo=0, scopeID=0)) internet. protocol. Factory: [info] RegionServer connection established (HOST:IPv6Addre ss(type= 'TCP', host=': :ffff:10. 245.222. 223', port=5251, flowInfo=0, scopeID=0) PEER:IPv6Addres s(type= 'TCP', host=': :ffff:10. 245.222. 182', port=34354, flowInfo=0, scopeID=0))
Apr 9 18:56:51 swoobat regiond[1297846]: regiond: [info] 127.0.0.1 POST /MAAS/api/
Apr 9 18:57:07 swoobat maas-log[1302692]: maas.interface: [info] eno1 (physical) on swoobat: New MAC, IP binding observed: 14:02:ec:3e:e1:78, 10.244.40.159
Apr 9 18:57:07 swoobat maas-log[1302692]: Apr 9 18:56:46 meinfoo maas.rpc.
Apr 9 18:57:07 swoobat maas-log[1302692]: Apr 9 18:57:07 meinfoo maas.interface: [info] eno1 (physical) on meinfoo: New MAC, IP binding observed: 14:02:ec:3e:e1:78, 10.244.40.159
Apr 9 18:57:07 swoobat maas-log[1302692]: Apr 9 18:57:07 leafeon maas.interface: [info] eno1 (physical) on leafeon: New MAC, IP binding observed: 14:02:ec:3e:e1:78, 10.244.40.159
Apr 9 18:57:15 swoobat rackd[1304815]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Addre
Apr 9 18:57:15 swoobat rackd[1304815]: Uninitialized: [info] ClusterClient connection established (HOST:IPv6Addre
Apr 9 18:57:15 swoobat rackd[1304815]: provisioningser
Apr 9 18:57:16 swoobat rackd[1304815]: provisioningser
Apr 9 18:57:16 swoobat maas-log[1302692]: Apr 9 18:57:16 meinfoo maas.rpc.
Apr 9 18:57:16 swoobat rackd[1304815]: provisioningser
Apr 9 18:57:16 swoobat rackd[1304815]: provisioningser
Apr 9 18:57:17 swoobat regiond[1297844]: twisted.
Apr 9 18:57:17 swoobat regiond[1297844]: twisted.
Apr 9 18:57:17 swoobat ...