Fuel for OpenStack

Logrotate config missed for ceph

Bug #1607117 reported by Max Lvov
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Sustaining
Fuel for OpenStack 10.0
8.0.x
Fix Released
High
Rodion Tikunov
Fuel for OpenStack 8.0-mu-4
Mitaka
Invalid
High
Fuel Sustaining
Fuel for OpenStack 9.1

Bug Description

MOS 8.0 MU1
3 controllers 5 computes 4 ceph 3 mongodb

Logs for ceph stored both on ceph nodes and controllers, but there is no logrotate config for ceph. Ceph during data redistribution (i.e. after new node addition) generates the overwhelming amount of logs and causes disk run out of free space (5Gb in 3 days).

Revision history for this message
Max Lvov (usrleon) wrote :

My proposal is to add /var/log/ceph/ to logrotate system and reduce loglevel to WARN

Alexander Kislitsky (akislitsky)
Changed in fuel:
milestone: none → 8.0-updates
assignee: nobody → MOS Maintenance (mos-maintenance)
Max Lvov (usrleon)
tags: added: customer-found
Anton Chevychalov (achevychalov)
Changed in fuel:
assignee: MOS Maintenance (mos-maintenance) → Anton Chevychalov (achevychalov)
Anton Chevychalov (achevychalov)
Changed in fuel:
status: New → Confirmed
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
importance: Undecided → High
Revision history for this message
Anton Chevychalov (achevychalov) wrote :

Problem with rotate exist in any release of MOS and in future release too.

If logs will not rotate in time it could stop node completely.

According to upstream (Ceph) it is common case to have INFO level of logs in /var/log/ceph so we will not change this to WARN.

According to upstream documentation (http://docs.ceph.com/docs/master/rados/troubleshooting/log-and-debug/#subsystem-log-and-debug-settings) we'd better accelerate log rotation in almost the same way as we do for Fuel. But we can't use same /usr/bin/fuel-logrotate half-hour job because it triggers syslog while we need to trigger ceph.

Oleksiy Molchanov (omolchanov)
tags: added: area-library
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/newton
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

9 and 10 are not affected. They have logrotate config coming from the package.

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Fuel Sustaining (fuel-sustaining-team)
Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

both milestones set to the Invalid due the previous comment

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Rodion Tikunov (rtikunov) wrote :

Proposed fix: https://review.fuel-infra.org/#/c/26636/

Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on 8.0 mu4 updates.
Ceph logrotate config file is present now. Logrotate works fine, managing old logs accoring to specified in config parameters.
https://paste.mirantis.net/show/11319/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.