logrotate doesn't work exited abnormally with [1] (0 was expected)

Bug #1455104 reported by Leontiy Istomin
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Opinion
High
Bartłomiej Piotrowski
6.0.x
Invalid
Undecided
Unassigned
8.0.x
Invalid
Undecided
Unassigned
StackLight
Fix Released
High
Swann Croiset
0.10
Fix Released
High
Swann Croiset
0.8
In Progress
High
Swann Croiset
0.9
Fix Committed
High
Swann Croiset

Bug Description

api: '1.0'
astute_sha: 055b2d82fe8499b27c7047295e2e36a7a2c5d430
auth_required: true
build_id: 2015-04-16_21-30-10
build_number: '317'
feature_groups:
- mirantis
fuellib_sha: db5f39e96e7ab9f79691202755e547bf8242661f
fuelmain_sha: 0de2d2039e76839d339f977df45b111bef7200d6
nailgun_sha: 52d92c86e68602fb5dd2f3b8870a773d20a387ef
openstack_version: 2014.2-6.1
ostf_sha: b0991dbad159f53d335efa5d31cb94016ad5312e
production: docker
python-fuelclient_sha: 279ffc358e40dbdc162cfe76898cbd0874529f1f
release: '6.1'

Successfully deployed the following configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-vlan,Ceph-all,Nova-debug,Nova-quotas, 6.1-317
Controllers:3 Computes:47

We have found that there is no archived logs on controller nodes:
root@node-1:~# ls -lah /var/log/*-all.log*
-rw-r----- 1 syslog syslog 279M May 14 13:59 /var/log/cinder-all.log
-rw-r----- 1 syslog syslog 356M May 14 13:59 /var/log/glance-all.log
-rw-r----- 1 syslog syslog 157M May 14 13:59 /var/log/heat-all.log
-rw-r----- 1 syslog syslog 1.8G May 14 13:59 /var/log/keystone-all.log
-rw-r----- 1 syslog syslog 3.3G May 14 13:59 /var/log/neutron-all.log
-rw-r----- 1 syslog syslog 718M May 14 13:59 /var/log/nova-all.log

For example cinder-all log hasn't been rotated:
root@node-1:~# head -1 /var/log/cinder-all.log
<159>May 12 22:41:11 node-1 cinder-manage backend <module 'cinder.db.sqlalchemy.migration' from '/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.pyc'>
root@node-1:~# tail -1 /var/log/cinder-all.log
<158>May 14 14:00:28 node-1 cinder-api 192.168.0.47 - - [14/May/2015 14:00:28] "OPTIONS / HTTP/1.0" 200 422 0.003460

root@node-1:~# ls -la /var/log/cinder/
total 416764
drwxr-x--- 2 cinder adm 4096 May 12 22:41 .
drwxrwxr-x 25 root syslog 4096 May 14 12:30 ..
-rw-r--r-- 1 cinder cinder 300518822 May 14 14:00 cinder-api.log
-rw-r--r-- 1 cinder cinder 709235 May 14 14:00 cinder-backup.log
-rw-r--r-- 1 cinder cinder 23014 May 12 22:41 cinder-manage.log
-rw-r--r-- 1 cinder cinder 29438943 May 14 14:00 cinder-scheduler.log
-rw-r--r-- 1 cinder cinder 96045179 May 14 14:00 cinder-volume.log

from messages log file:
<13>May 14 13:50:01 node-1 logrotate: ALERT exited abnormally with [1] (0 was expected)
We can see this alert every 10 minutes

Diagnostic snapshot is here http://mos-scale-share.mirantis.com/fuel-snapshot-2015-05-14_08-14-00.tar.xz

no longer affects: mos
Changed in fuel:
assignee: nobody → Bartlomiej Piotrowski (bpiotrowski)
milestone: none → 6.1
importance: Undecided → High
Revision history for this message
Bartłomiej Piotrowski (bpiotrowski) wrote :

It's a really old iso. There were a few changes to logrotate configuration in the meantime:

https://github.com/stackforge/fuel-library/commit/c8702d93371e138e8ed362cf25e959ce960d79f3
https://github.com/stackforge/fuel-library/commit/86947bb9df42b7f3bd8625be456ffe2dafdc50ee
https://github.com/stackforge/fuel-library/commit/eb3e29caec4c2979d22e46a470610e8f6b8ac751

If you think that the issue is still present, please post here the logs from logrotate runs, because I don't think those are included in diagnostic snapshot.

1. logrotate -v /etc/logrotate.conf
2. Edit our logrotate shell wrapper to include 'set -x' before anything gets executed and attach the output of run.

Changed in fuel:
status: New → Incomplete
Changed in fuel:
status: Incomplete → Invalid
Revision history for this message
Serg Lystopad (slystopad) wrote :

We still see the bug in Fuel 8.0
cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

Changed in fuel:
status: Invalid → Opinion
tags: added: customer-found
Revision history for this message
Stanislav Kolenkin (skolenkin) wrote :

The crontab contain the following (crontab -l) on compute nodes:
# HEADER: This file was autogenerated at 2016-07-26 22:11:14 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: fuel-logrotate
*/30 * * * * /usr/bin/fuel-logrotate
# Puppet Name: lma_collector logrotate
*/30 * * * * /usr/local/bin/logrotate_lma_collector

Following workaround may help here.
Please, update crontab file "/var/spool/cron/crontabs/root” as follows.
Substitute these lines:
*/30 * * * * /usr/bin/fuel-logrotate
# Puppet Name: lma_collector logrotate
*/30 * * * * /usr/local/bin/logrotate_lma_collector
with:
25 * * * * /usr/bin/fuel-logrotate
# Puppet Name: lma_collector logrotate
40 * * * * /usr/local/bin/logrotate_lma_collector

Swann Croiset (swann-w)
Changed in lma-toolchain:
importance: Undecided → High
assignee: nobody → LMA-Toolchain Fuel Plugins (mos-lma-toolchain)
status: New → Confirmed
Revision history for this message
Swann Croiset (swann-w) wrote :

This is the root cause when concurrent execution of logrotate occur:

error: error renaming temp state file /var/lib/logrotate/status.tmp

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (master)

Fix proposed to branch: master
Review: https://review.openstack.org/355949

Changed in lma-toolchain:
assignee: LMA-Toolchain Fuel Plugins (mos-lma-toolchain) → Swann Croiset (swann-w)
status: Confirmed → In Progress
Revision history for this message
Dmitry Guryanov (dguryanov) wrote :

Are there any fixes needed on the fuel side?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (master)

Reviewed: https://review.openstack.org/355949
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=99508dbc3919863330d9a879529b330fdd1d5dff
Submitter: Jenkins
Branch: master

commit 99508dbc3919863330d9a879529b330fdd1d5dff
Author: Swann Croiset <email address hidden>
Date: Tue Aug 16 15:50:55 2016 +0200

    Fixes the concurrent execution of logrotate

    Change-Id: I1546744d3c4cab8fc359b6571d99aabde3c3ff2d
    Closes-bug: #1455104

Changed in lma-toolchain:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (stable/0.10)

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/356047

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (stable/0.9)

Fix proposed to branch: stable/0.9
Review: https://review.openstack.org/356048

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (stable/0.8)

Fix proposed to branch: stable/0.8
Review: https://review.openstack.org/356050

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (stable/0.10)

Reviewed: https://review.openstack.org/356047
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=43a6ad6a79c0644724693541a6cc2d74c2e3dadd
Submitter: Jenkins
Branch: stable/0.10

commit 43a6ad6a79c0644724693541a6cc2d74c2e3dadd
Author: Swann Croiset <email address hidden>
Date: Tue Aug 16 15:50:55 2016 +0200

    Fixes the concurrent execution of logrotate

    Change-Id: I1546744d3c4cab8fc359b6571d99aabde3c3ff2d
    Closes-bug: #1455104
    (cherry picked from commit 99508dbc3919863330d9a879529b330fdd1d5dff)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (stable/0.9)

Reviewed: https://review.openstack.org/356048
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=7ee4a5b53b8a0f167076fba8351035a492f2e66c
Submitter: Jenkins
Branch: stable/0.9

commit 7ee4a5b53b8a0f167076fba8351035a492f2e66c
Author: Swann Croiset <email address hidden>
Date: Tue Aug 16 15:50:55 2016 +0200

    Fixes the concurrent execution of logrotate

    Change-Id: I1546744d3c4cab8fc359b6571d99aabde3c3ff2d
    Closes-bug: #1455104
    (cherry picked from commit 99508dbc3919863330d9a879529b330fdd1d5dff)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-plugin-lma-collector (stable/0.8)

Change abandoned by Swann Croiset (<email address hidden>) on branch: stable/0.8
Review: https://review.openstack.org/356050
Reason: no body wants it :/

no longer affects: lma-toolchain/1.0
Changed in lma-toolchain:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.