Post some debug log from 3.4.0-beta3, from https://oil-jenkins.canonical.com/artifacts/6616e90f-975d-4443-9fe7-bbc27fa62bb8/generated/generated/maas/logs-2023-06-15-14.53.09.tgz. Very close to what we have in #1.
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 RegionServer,29,::ffff:10.245.222.215: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.245.222.204', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.245.222.215', port=45764, flowInfo=0, scopeID=0)) Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver.ipc: [info] Worker pid:1523454 lost RPC connection to ('tmhdw8', '10.245.222.204', 5250). Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 RegionServer,30,::ffff:10.245.222.215: [info] RegionServer connection lost (HOST:IPv6Address(type='TCP', host='::ffff:10.245.222.204', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.245.222.215', port=45766, flowInfo=0, scopeID=0)) Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver: [error] ################################ Exception: Connection was closed cleanly. ################################ Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver: [error] Traceback (most recent call last): Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response Jun 15 14:52:21 sunset sh[1523069]: response = wrapped_callback(request, *callback_args, **callback_kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 293, in view_atomic_with_post_commit_savepoint Jun 15 14:52:21 sunset sh[1523069]: return view_atomic(*args, **kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3.10/contextlib.py", line 79, in inner Jun 15 14:52:21 sunset sh[1523069]: return func(*args, **kwds) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 62, in __call__ Jun 15 14:52:21 sunset sh[1523069]: response = super().__call__(request, *args, **kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/django/views/decorators/vary.py", line 20, in inner_func Jun 15 14:52:21 sunset sh[1523069]: response = func(*args, **kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3.10/dist-packages/piston3/resource.py", line 197, in __call__ Jun 15 14:52:21 sunset sh[1523069]: result = self.error_handler(e, request, meth, em_format) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3.10/dist-packages/piston3/resource.py", line 195, in __call__ Jun 15 14:52:21 sunset sh[1523069]: result = meth(request, *args, **kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 371, in dispatch Jun 15 14:52:21 sunset sh[1523069]: return function(self, request, *args, **kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/maasserver/api/support.py", line 217, in wrapper Jun 15 14:52:21 sunset sh[1523069]: return func(self, request, *args, **kwargs) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/maasserver/api/pods.py", line 346, in compose Jun 15 14:52:21 sunset sh[1523069]: machine = form.compose() Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/maasserver/forms/pods.py", line 741, in compose Jun 15 14:52:21 sunset sh[1523069]: ).wait(timeout) Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 198, in wait Jun 15 14:52:21 sunset sh[1523069]: result.raiseException() Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException Jun 15 14:52:21 sunset sh[1523069]: raise self.value.with_traceback(self.tb) Jun 15 14:52:21 sunset sh[1523069]: twisted.internet.error.ConnectionDone: Connection was closed cleanly. Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver.ipc: [info] Worker pid:1523454 lost RPC connection to ('tmhdw8', '10.245.222.204', 5250).
Post some debug log from 3.4.0-beta3, from https:/ /oil-jenkins. canonical. com/artifacts/ 6616e90f- 975d-4443- 9fe7-bbc27fa62b b8/generated/ generated/ maas/logs- 2023-06- 15-14.53. 09.tgz.
Very close to what we have in #1.
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 RegionServer, 29,::ffff: 10.245. 222.215: [info] RegionServer connection lost (HOST:IPv6Addre ss(type= 'TCP', host=': :ffff:10. 245.222. 204', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Addres s(type= 'TCP', host=': :ffff:10. 245.222. 215', port=45764, flowInfo=0, scopeID=0)) 30,::ffff: 10.245. 222.215: [info] RegionServer connection lost (HOST:IPv6Addre ss(type= 'TCP', host=': :ffff:10. 245.222. 204', port=5250, flowInfo=0, scopeID=0) PEER:IPv6Addres s(type= 'TCP', host=': :ffff:10. 245.222. 215', port=45766, flowInfo=0, scopeID=0)) ####### ####### ####### #### Exception: Connection was closed cleanly. ####### ####### ####### ####### #### python3/ dist-packages/ django/ core/handlers/ base.py" , line 181, in _get_response callback( request, *callback_args, **callback_kwargs) python3/ dist-packages/ maasserver/ utils/views. py", line 293, in view_atomic_ with_post_ commit_ savepoint python3. 10/contextlib. py", line 79, in inner python3/ dist-packages/ maasserver/ api/support. py", line 62, in __call__ .__call_ _(request, *args, **kwargs) python3/ dist-packages/ django/ views/decorator s/vary. py", line 20, in inner_func python3. 10/dist- packages/ piston3/ resource. py", line 197, in __call__ handler( e, request, meth, em_format) python3. 10/dist- packages/ piston3/ resource. py", line 195, in __call__ python3/ dist-packages/ maasserver/ api/support. py", line 371, in dispatch python3/ dist-packages/ maasserver/ api/support. py", line 217, in wrapper python3/ dist-packages/ maasserver/ api/pods. py", line 346, in compose python3/ dist-packages/ maasserver/ forms/pods. py", line 741, in compose python3/ dist-packages/ crochet/ _eventloop. py", line 198, in wait raiseException( ) python3/ dist-packages/ twisted/ python/ failure. py", line 475, in raiseException with_traceback( self.tb) internet. error.Connectio nDone: Connection was closed cleanly.
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver.ipc: [info] Worker pid:1523454 lost RPC connection to ('tmhdw8', '10.245.222.204', 5250).
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 RegionServer,
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver: [error] #######
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver: [error] Traceback (most recent call last):
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: response = wrapped_
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: return view_atomic(*args, **kwargs)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: return func(*args, **kwds)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: response = super()
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: response = func(*args, **kwargs)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: result = self.error_
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: result = meth(request, *args, **kwargs)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: return function(self, request, *args, **kwargs)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: return func(self, request, *args, **kwargs)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: machine = form.compose()
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: ).wait(timeout)
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: result.
Jun 15 14:52:21 sunset sh[1523069]: File "/usr/lib/
Jun 15 14:52:21 sunset sh[1523069]: raise self.value.
Jun 15 14:52:21 sunset sh[1523069]: twisted.
Jun 15 14:52:21 sunset sh[1523069]: 2023-06-15 14:52:21 maasserver.ipc: [info] Worker pid:1523454 lost RPC connection to ('tmhdw8', '10.245.222.204', 5250).