Rack keeps opening idle RPC connections over the limits

Bug #2032949 reported by Jacopo Rota
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Status tracked in 3.5
3.3
New
Undecided
Unassigned
3.4
Triaged
Undecided
Unassigned
3.5
Triaged
Undecided
Unassigned

Bug Description

As per title, the rack is opening idle connections without limits under no load.

Steps to reproduce:

1) add a log statement like
```
        log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py$_doUpdate (at first line of the method is fine)
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:

```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```

and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are

```
    max_idle_rpc_connections = ConfigurationOption(
        "max_idle_rpc_connections",
        "The nominal number of connections to have per endpoint",
        Number(min=1, max=1024, if_missing=1),
    )
    max_rpc_connections = ConfigurationOption(
        "max_rpc_connections",
        "The maximum number of connections to scale to when under load",
        Number(min=1, max=1024, if_missing=4),
    )
```

If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.

Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see as many disconnect messages as the number of open connections in the rack

Jacopo Rota (r00ta)
summary: - Rack keeps opening idle connections over the limits
+ Rack keeps opening idle RPC connections over the limits
Jacopo Rota (r00ta)
description: updated
description: updated
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.