Wrong symlinks for "bak" directories cause logrotate failures

Raised to high as this issue with symlinks affects the cloud operations in a long term

I reassigned this bug to Nailgun team as it seems the duplicating symlinks is being created by nailgun backend after environment reset/recreate.

FYI, in case it makes any difference behind the hood.

In the case that lead to this bugreport (https://bugs.launchpad.net/fuel/+bug/1427197) the environment was not reset. We deployed controllers and discovered that Telemetry node(s) could not be deployed from the GUI afterwards. So the controllers were deleted an added again.

I reproduced with bug again with iso #153.
It is enough to remove a node from environment and add another one. There is a chance what the new node received the same name as it had been given to the removed node. And for such case, the symlink created for removed node will point to wrong directory: http://paste.openstack.org/show/189207/

/var/log/remote/10.108.0.3 -> node-1.test.domain.local
/var/log/remote/10.108.0.7 -> node-1.test.domain.local

But the first one should point to the node-1.test.domain.local.bak dir instead

And here is a test case for this bug:
1) Remove a primary-controller node from environment, and deploy
2) Add a new controller, and deploy
3) repeat the steps 1-2 until some of the old node name swill be reused for a new node
4) verify output of ls -ld /var/log/remote/* | awk '{ print $11 }' | sort | uniq -d
it should give nothing (test passed), otherwise it will print the names of affected nodes

I takes ages for me to deploy anything, I need to wait until we have our own office server for tests to even reproduce this bug :/

It seems, the symlinks are created here https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/task/helpers.py#L64-L125

@Ivan, do you have this bug in progress? If so, could you please update the status?

@Bogdan, Ivan is on vacation. I think someone else should work on this.

It probably will be addressed, if we will not have .bak at all:
https://bugs.launchpad.net/fuel/+bug/1428825

Probably, we can address this issue:
 error: error creating output file /var/log/remote/172.31.2.82/mongod.27017.log.1: File exists
By using:
dateformat -%Y%m%d-%s

So, rotated files will look like:
filename.log-20150412-1428814861.gz

Since unix time seconds is uniq every time, no 'File exists' errors should be present.
This need to be tested.

IMHO, the prepare_syslog_dir() should be fixed to not create duplicating sym links pointing to the same directories AND must be covered by tests - there is none yet for this method

This bug will be fix by patchset for https://bugs.launchpad.net/fuel/+bug/1428825 becaus it remove also logs for deleted nodes, so there shodn't be a situation like this described in bug

@Kamil, this bug is not about environment deletion. The subject is about removing and adding nodes to existing environments. Please confirm that the issue https://bugs.launchpad.net/fuel/+bug/1428825 would have resolved this one as well. Until then I removed the duplicate status. Thank you!

I talk with Sylwester and we will remove old logs when you remove node from cluster.

Reproduced with ISO #414
[root@nailgun ~]# ls -ld /var/log/remote/* | awk '{ print $11 }' | sort | uniq -d
node-3.test.domain.local
node-4.test.domain.local

as a result logrotate fails with "File already exists" error mentioned above.

How to reproduce:
1) Deploy 1 controller
2) Add 2 controllers and deploy
3) Add 2 more controllers and deploy
4) Remove 2 controllers and deploy
5) Reset env and repeat steps 1-4 several times

Please provide a snapshot, look like a bug in previous patch.

I don't have this environment anymore, sorry

@Lukasz, there is a scale labs affected by this issue permanently due to the scecific flow they use for add/remove reset envs - feel free to investigate. You can contact Leontiy Istomin to get the environment

On lab http://172.16.44.10/ there are old iso (build number: 317) without fix of this problem. We still need reproduce this bug on newer with fix for this problem (iso >383). I change status back for incomplite

6.1-425. I deployed an env then deleteed it then created and deployed new env.

[root@fuel ~]# ls -ld /var/log/remote/* | awk '{ print $11 }' | sort | uniq -d

[root@fuel ~]#

It seems it works fine. Should I check something else to be sure?

Looks like this bug fixed with this bugfix: https://bugs.launchpad.net/fuel/+bug/1428825

reproduced the issue with 6.1-511 build:
http://paste.openstack.org/show/278338/
[root@fuel ~]# du -hs /var/log/docker-logs/remote/node-39.domain.tld/* | grep G
5.4G /var/log/docker-logs/remote/node-39.domain.tld/keystone-all.log
7.4G /var/log/docker-logs/remote/node-39.domain.tld/neutron-server.log
1.4G /var/log/docker-logs/remote/node-39.domain.tld/nova-api.log

@Leontiy, and what was an output for the
ls -ld /var/log/remote/* | awk '{ print $11 }' | sort | uniq -d
?

@Bogdan, you can find requested info here: http://paste.openstack.org/show/278338/

Yes, thank you, didn't notice. This is the case, indeed.

The fix is ready, now I need to test it to be sure that this issue will not happen when reseting env. It should be done tomorrow morning (CEST)

Fix proposed to branch: master
Review: https://review.openstack.org/190965

Reviewed: https://review.openstack.org/190965
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f39b95b6dc993ba1224ddfab3d1e8ea0848f87d7
Submitter: Jenkins
Branch: master

commit f39b95b6dc993ba1224ddfab3d1e8ea0848f87d7
Author: Sebastian Kalinowski <email address hidden>
Date: Fri Jun 12 09:36:24 2015 +0200

    Remove old logs when resetting environment

    If logs are left, they cause failing of logratate since
    there are hanging symlinks left.
    It was fixed for removing nodes and fix for resetting must be added.

    Change-Id: I047bf4161cf1f76b30bf7a0f5b6d13207d314019
    Closes-Bug: #1428150

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/191001

Reviewed: https://review.openstack.org/191001
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=86af2b9bf3b9998e576bc5e8c8c7f02cfb49b492
Submitter: Jenkins
Branch: stable/6.1

commit 86af2b9bf3b9998e576bc5e8c8c7f02cfb49b492
Author: Sebastian Kalinowski <email address hidden>
Date: Fri Jun 12 09:36:24 2015 +0200

    Remove old logs when resetting environment

    If logs are left, they cause failing of logratate since
    there are hanging symlinks left.
    It was fixed for removing nodes and fix for resetting must be added.

    Change-Id: I047bf4161cf1f76b30bf7a0f5b6d13207d314019
    Closes-Bug: #1428150
    (cherry picked from commit f39b95b6dc993ba1224ddfab3d1e8ea0848f87d7)

Won't Fix for 6.0-updates as we have no delivery channel for Fuel fixes in 6.0