Ubuntu logs are incomplete in snapshot

Already fixed (see dup issue)
Make sure CI gate includes https://review.openstack.org/#/c/85970/ astute fix

The patch you link to is merged at 11 Apr. But Fuel CI used fuel-5.0-107 ISO which was build from master yesterday (16 Apr).

This means the patch is already there, but logs are still incomplete.

The 2nd patch addressed for the same issue wasn't merged yet https://review.openstack.org/#/c/85975. I hope, once merged - it should resolve the issue completely

Logs snapshot now provide puppet logs from the local nodes as well.
Since that, this issue is not a blocker for logs analyzing in CI jobs.
That also allows to compare the "cut" logs from /remote destination vs local ones

Almost all Ubuntu CI gate failures are affected by this issue, example https://fuel-jenkins.mirantis.com/job/master_fuellib_review_systest_ubuntu/1251/

Here you can compare the local and remote logs and analyze the possible root cause of such cutting it off

As far as I can see all logs is missing in snapshot starting from the "cut moment", so the issue is related to all remote logs delivery not only the puppet ones

run_user:run_group should be syslog:syslog for Ubuntu. That is a root cause of missing logs (in Fuel, all syslog templates are for root:root)

Fix proposed to branch: master
Review: https://review.openstack.org/102870

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/103076

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/103095

Reviewed: https://review.openstack.org/103076
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0271711fafb3c558a41e84c37c6dadb571e59619
Submitter: Jenkins
Branch: stable/5.0

commit 0271711fafb3c558a41e84c37c6dadb571e59619
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jun 26 19:02:02 2014 +0300

    Use syslog:syslog for Ubuntu

    In Fuel, all syslog templates are for root:root. But
    run_user:run_group should be syslog:syslog for Ubuntu.

    poke ci
    Closes-bug: #1308684

    Change-Id: I099560f22b409cf60a6c565af8a1ab13bac48a7c
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Should this be documented as a Fixed Issue in the Release Notes? If so, did you fix the owner/group permissions for syslog on Ubuntu or do users need to configure this on the Settings page?

AFAIK, the issue was not introduced in the previous releases. It was fixed for 5.0.1 only, other targets are on review. So doc impact seems zero.

Reviewed: https://review.openstack.org/102870
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ffd87c7a17d2f21e096b9f6462fd9a9f3820cd2b
Submitter: Jenkins
Branch: master

commit ffd87c7a17d2f21e096b9f6462fd9a9f3820cd2b
Author: Bogdan Dobrelya <email address hidden>
Date: Thu Jun 26 19:02:02 2014 +0300

    Use syslog:syslog for Ubuntu

    In Fuel, all syslog templates are for root:root. But
    run_user:run_group should be syslog:syslog for Ubuntu.

    Closes-bug: #1308684

    Change-Id: I099560f22b409cf60a6c565af8a1ab13bac48a7c
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Additional fix required for syslog files created by the custom logging rules ,e.g.:
-rw-r--r-- 1 root root 21K Jul 16 00:24 /var/log/alternatives.log
-rw-r--r-- 1 root root 532 Jul 16 00:16 /var/log/boot.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/ceilometer-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/cinder-all.log
-rw-r--r-- 1 root root 0 Jul 16 00:19 /var/log/corosync.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/cron.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/daemon.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/dashboard.log
-rw-r--r-- 1 root root 478K Jul 16 00:36 /var/log/dpkg.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/glance-all.log
-rw-r----- 1 root adm 0 Jul 16 08:31 /var/log/haproxy.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/heat-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/keystone-all.log
-rw-r--r-- 1 root root 1.5M Jul 16 08:35 /var/log/mcollective.log
-rw-r--r-- 1 root root 149 Jul 16 00:24 /var/log/memcached.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/murano-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/mysqld.log
-rw-r--r-- 1 root root 12M Jul 16 08:35 /var/log/nailgun-agent.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/neutron-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/nova-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/pacemaker.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/sahara-all.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/user.log
-rw-r----- 1 root adm 0 Jul 16 00:18 /var/log/zabbix_server.log

Such files will be locked for syslog once he changed its running user

Note that since the first merged patch successfully resolves *remote* logging issues for 5.0.1 and 5.1, we could lower the issue to high (remote logs OK, only its local copies affected). But 4.1.x is still requiring both patches, thus, staying at critical

Fix proposed to branch: master
Review: https://review.openstack.org/107273

How to test the patchset:
At deployed OSt nodes with Ubuntu, the output for "find /var/log/ -type f -size 0 -user root -group adm | xargs -n1 ls -lh"
should be empty

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/107294

Reviewed: https://review.openstack.org/107273
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c3efecb85c45b9f4987163ea8bff737bf317762e
Submitter: Jenkins
Branch: master

commit c3efecb85c45b9f4987163ea8bff737bf317762e
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 16 12:27:30 2014 +0300

    Update logfile ownership for syslog in Ubuntu

    With $FileGroup:$FileOwner = root:adm, we would have some
    syslog logfiles inaccessible then syslog running user would
    be changed from root to syslog.
    root:adm should not be set for syslog logfiles since
    we use syslog:syslog as drop-provilege-to as well
    (http://www.rsyslog.com/doc/droppriv.html)

    Closes-bug: #1308684

    Change-Id: I4ab98cd77d29d986d202051b7e66c8abb13e4c15
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Reviewed: https://review.openstack.org/107294
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3315d10e113c3778c1f35c0c90ea88e9da92dcf8
Submitter: Jenkins
Branch: stable/5.0

commit 3315d10e113c3778c1f35c0c90ea88e9da92dcf8
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jul 16 12:27:30 2014 +0300

    Update logfile ownership for syslog in Ubuntu

    With $FileGroup:$FileOwner = root:adm, we would have some
    syslog logfiles inaccessible then syslog running user would
    be changed from root to syslog.
    root:adm should not be set for syslog logfiles since
    we use syslog:syslog as drop-provilege-to as well
    (http://www.rsyslog.com/doc/droppriv.html)

    Closes-bug: #1308684

    Change-Id: I4ab98cd77d29d986d202051b7e66c8abb13e4c15
    Signed-off-by: Bogdan Dobrelya <email address hidden>

If there would be 4.1.2 by any chances, update the 4.1.2 milestone target status as in progress as well

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/4.1
Review: https://review.openstack.org/103095

Fix released:
api: '1.0'
astute_sha: f655ee86ebf0359b014f00cff63d0aaf15c65308
auth_required: true
build_id: 2014-08-01_10-37-32
build_number: '382'
feature_groups:
- mirantis
fuellib_sha: 5571b86a667e28d4c9770fcce4d32163dee5a710
fuelmain_sha: 7990f5bfa7fea5b74ebf0402b1918109b9bc505b
nailgun_sha: 51f32395eebe2514e78eb7e0a85e694826be40d6
ostf_sha: 15f3be5fbafb7a8c7075b5077a5074a50e679c25
production: docker
release: '5.1'

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/4.1
Review: https://review.openstack.org/103095

CI problems are not relevant anymore. Closing as Won't fix.