Environment:
MOS 6.0, HA (reproduced with both 1 and 3 controllers), Ceilometer is enabled.
In 6.0, Keystone is configured to send notifications about it's activity to Ceilometer via MQ. (See notification_driver setting in keystone.conf). Due to it's nature (identity service), Keystone generates a huge stream of notifications. Almost all of them are "identity.authenticate.success". This creates additional load on the DB and can lead to poor performance. We have encountered this when Ceilometer-related OSTF and Tempest tests failed with timeout.
After ~20 hours after the deployment meter-list contained over 60k entries:
root@node-2:~# ceilometer meter-list | wc -l
62670
99.9% of them were keystone's "identity.authenticate.success". The cluster had 3 controllers and 2 computes. It mostly has been idle (no OSTF or Tempest runs; manually booted 4 instances).
The workaround:
1. Comment out notification_driver=messaging in keystone.conf on all controllers; restart keystone. This will stop it from sending metrics.
2. Configure DB entries TTL in ceilometer. Set time_to_live to preferred TTL is seconds in ceilometer.conf.
3. Manually run ceilometer-expirer to delete outdated data.
I think it's High priority issue as Ceilometer DB gets polluted. At the end, Ceilometer won't be functional due to DB overload. If we can avoid spam from Keystone - let's do it.