| 2023-08-24 12:53:59 |
Jacopo Rota |
bug |
|
|
added bug |
| 2023-08-24 12:54:07 |
Jacopo Rota |
summary |
Rack keeps opening idle connections over the limits |
Rack keeps opening idle RPC connections over the limits |
|
| 2023-08-24 12:58:03 |
Jacopo Rota |
nominated for series |
|
maas/3.5 |
|
| 2023-08-24 12:58:03 |
Jacopo Rota |
bug task added |
|
maas/3.5 |
|
| 2023-08-24 12:58:14 |
Jacopo Rota |
nominated for series |
|
maas/3.3 |
|
| 2023-08-24 12:58:14 |
Jacopo Rota |
bug task added |
|
maas/3.3 |
|
| 2023-08-24 12:58:14 |
Jacopo Rota |
nominated for series |
|
maas/3.4 |
|
| 2023-08-24 12:58:14 |
Jacopo Rota |
bug task added |
|
maas/3.4 |
|
| 2023-08-24 13:01:59 |
Jacopo Rota |
description |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections. |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see a lot of disconnect messages in the regiond logs |
|
| 2023-08-24 13:02:35 |
Jacopo Rota |
description |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see a lot of disconnect messages in the regiond logs |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see as many disconnect messages as the number of open connections in the rack |
|
| 2023-08-24 13:04:03 |
Jacopo Rota |
description |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see as many disconnect messages as the number of open connections in the rack |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py$_doUpdate (as first line of the method is fine)
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see as many disconnect messages as the number of open connections in the rack |
|
| 2023-08-24 13:04:17 |
Jacopo Rota |
description |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py$_doUpdate (as first line of the method is fine)
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see as many disconnect messages as the number of open connections in the rack |
As per title, the rack is opening idle connections without limits under no load.
Steps to reproduce:
1) add a log statement like
```
log.info("THERE ARE " + str(len(self.getAllClients())) + " CONNECTIONS!")
```
in provisioningserver/rpc/clusterservice.py$_doUpdate (at first line of the method is fine)
2) install maas from sources
3) maas init region+rack
4) wait **some minutes** and take a look at the logs:
```
Aug 24 12:35:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 0 CONNECTIONS!
Aug 24 12:35:20 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 3
...
CONNECTIONS!
Aug 24 12:35:45 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:46 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:47 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:48 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:35:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 4 CONNECTIONS!
Aug 24 12:36:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 6 CONNECTIONS!
Aug 24 12:36:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 8 CONNECTIONS!
Aug 24 12:37:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 10 CONNECTIONS!
Aug 24 12:37:49 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 12 CONNECTIONS!
Aug 24 12:38:19 maas-edge maas-rackd[10519]: provisioningserver.rpc.clusterservice: [info] THERE ARE 14 CONNECTIONS!
```
and they keep increasing every 30 seconds. In addition to this strange behaviour, it is actually exceeding the configuration limits which are
```
max_idle_rpc_connections = ConfigurationOption(
"max_idle_rpc_connections",
"The nominal number of connections to have per endpoint",
Number(min=1, max=1024, if_missing=1),
)
max_rpc_connections = ConfigurationOption(
"max_rpc_connections",
"The maximum number of connections to scale to when under load",
Number(min=1, max=1024, if_missing=4),
)
```
If the region has 4 workers, it reaches 20 connections and it stops. If I specify the max max_rpc_connections=1000, then it keeps opening connections.
Note that they are all open connections. As a matter of facts, in the regiond logs there are no "disconnect" messages and if you kill the rackd process you will see as many disconnect messages as the number of open connections in the rack |
|
| 2023-08-24 15:52:27 |
Jacopo Rota |
maas/3.4: status |
New |
Triaged |
|
