Fuel for OpenStack

Ceph OSD logs are not rotated

Bug #1675759 reported by Denis Klepikov
This bug report is a duplicate of:  Bug #1607117: Logrotate config missed for ceph. Edit Remove
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Alexey Stupnikov
8.0.x
Confirmed
High
Alexey Stupnikov
Fuel for OpenStack 8.0-mu-4
Mitaka
Invalid
High
Alexey Stupnikov
Fuel for OpenStack 9.2-mu-2

Bug Description

Ceph OSD logs are not rotated on ceph-OSD nodes.

root@node-8:/var/log/ceph# cat /etc/logrotate.d/fuel.nodaily
# managed by puppet

"/var/log/aodh/*.log"
"/var/log/audit/audit.log"
"/var/log/ceilometer/*.log"
"/var/log/cinder/*.log"
"/var/log/glance/*.log"
"/var/log/heat/*.log"
"/var/log/keystone/*.log"
"/var/log/murano/*.log"
"/var/log/neutron/*.log"
"/var/log/nova/*.log"
"/var/log/sahara/*.log"
"/var/log/swift/*.log"
"/var/log/*-all.log"
"/var/log/auth.log"
"/var/log/corosync.log"
"/var/log/cron.log"
"/var/log/daemon.log"
"/var/log/dashboard.log"
"/var/log/debug"
"/var/log/kern.log"
"/var/log/mail.log"
"/var/log/messages"
"/var/log/mongod.log"
"/var/log/mysqld.log"
"/var/log/nailgun-agent.log"
"/var/log/pacemaker.log"
"/var/log/sudo.log"
"/var/log/syslog"
"/var/log/user.log"
"/var/log/upstart/*.log"
"/var/log/remote/*.log"
...

root@node-8:/var/log/ceph# ls -la /var/log/ceph/
total 388
drwxr-xr-x 2 root root 4096 Mar 24 11:51 .
drwxrwxr-x. 16 root syslog 4096 Mar 24 11:51 ..
-rw-r--r-- 1 root root 189164 Mar 24 12:19 ceph-osd.0.log
-rw-r--r-- 1 root root 194158 Mar 24 12:19 ceph-osd.1.log

During long environment usage /var/log/ceph/*.log files can use a lot of disk space and cause cluster outage due to lack of the free space.

/var/log/ceph/*.log PATH should be added to /etc/logrotate.d/fuel.nodaily config file.

Workaround:
 Open /etc/logrotate.d/fuel.nodaily config file and add "/var/log/ceph/*.log"

...
"/var/log/aodh/*.log"
"/var/log/audit/audit.log"
"/var/log/ceilometer/*.log"
"/var/log/ceph/*.log"
"/var/log/cinder/*.log"
"/var/log/glance/*.log"
"/var/log/heat/*.log"
"/var/log/keystone/*.log"
"/var/log/murano/*.log"
...

and save this file.

New configuration will be applied during the next fuel-logrotate run (see the `crontab -l` output )

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

I suggest to create dedicated logrotate configuration file for ceph and to ship it with ceph packages. IMO there is no reason to use general fuel logrotate job.

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Sorry, actually there is a reason: you probably shouldn't update ceph packages unless something is broken.

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

According to @kdanilov, a logrotation was implemented in ceph (though it could not be implemented in old versions) and doesn't require additional configuration. Have to doublecheck everything.

Alexey Stupnikov (astupnikov)
Changed in fuel:
assignee: Fuel for Openstack (fuel) → Alexey Stupnikov (astupnikov)
Revision history for this message
Denis Klepikov (dklepikov) wrote :

Also current /etc/logrotate.d/ceph configuration can be affected by conflict of "sharedscripts" and "delaycompress" options

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

1. Regarding possible conflict between sharedscripts and delaycompress
   configuration options:

   - sharedscripts is used to run postrotate scripts only when all log
     files are rotated. It allows us, for example, to not restart same
     daemon multiple times.
   - delaycompress option is used to store a latest rotated logfile as
     is (in other words to not compress latest rotated file)

   I don't see how is it possible to organize negative interaction of
   those features; I couldn't find anything on google. So we need some
   steps to reproduce to proceed with this issue.

2. Regarding missing logrotate configuration:

   - MOS10. cephv10.2.6 (jewel)

     No delaycompress statement in ceph's configuration.

     We use ubuntu's repo to install ceph packages. Logrotate file
     shipped with the package is aligined with upstream. There are no
     logrotate-related bugs reported that can cause reported issue.

   - MOS9.2. cephv0.94.9 (hammer)

     No delaycompress statement in ceph's configuration.

     We ship dedicated logrotate configuration with ceph package.
     Community ceph ships similar logrotate config with the same
     package. There are no reported issues with community ceph
     logrotate, all later fixes were used to optimize something or
     were centos-related.

   - MOS8. cephv0.94.5 (hammer)

     Duplicate of bug #1607117

Changed in fuel:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.