mariadbcheck@ service is called with uniq parameter every time causing high cardinality
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack-Ansible |
Triaged
|
Medium
|
Unassigned |
Bug Description
There is a systemd service template mariadbcheck@ which is been called with unique parameter every time.
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: Starting mariadbcheck@ service (10.33.
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: mariadbcheck@
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: Finished mariadbcheck@ service (10.33.
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: mariadbcheck@
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: Finished mariadbcheck@ service (10.33.
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: mariadbcheck@
Aug 08 12:03:44 infra2-osa-mvp systemd[1]: Finished mariadbcheck@ service (10.33.
The problem is that this 'tail' is creating a new UNIT value for journald, prometheus with node-exporter with enabled systemd-collector and for loki, if systemd logs are properly parsed.
In five days it generated about a third of million of unique values for the service name:
time journalctl -F UNIT|wc -l
311751
real 1m6.551s
user 0m11.799s
sys 0m51.035s
I believe, having O(t) systemd unit names will break many monitoring systems (because there is an assumption that unit list stays more or less the same).
Proposition: remove timestamp from parametrization for calling mariadbcheck@ service.
Hi,
Eventually, it is required to have templated name for the service, that is followed by the socket with Accept=yes /www.freedeskto p.org/software/ systemd/ man/systemd. socket. html#Accept=
https:/
In order to overcome this issue, we'd need to disable `Accept`, which requires to have a proper daemon behind it, rather then just a script, that was moved from xinetd.
At the moment we don't have time to refactor clustercheck [1] as we wanted to work on replacing HAproxy with ProxySQL for MariaDB balancing.
So contributions to that in the meanwhile are welcome.
[1] https:/ /opendev. org/openstack/ openstack- ansible- galera_ server/ src/branch/ master/ templates/ clustercheck. j2