Fuel master disk can fill because logrotate errors caused by symlinks and pattern match in /var/log/remote
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Bogdan Dobrelya | ||
4.1.x |
Fix Committed
|
High
|
Bogdan Dobrelya | ||
5.0.x |
Fix Released
|
High
|
Bogdan Dobrelya |
Bug Description
The Fuel logrotate config /etc/logrotate.
/var/log/
In /var/log/remote, there are symlinks created for each node
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.3 -> node-1.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.4 -> node-2.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.5 -> node-3.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.6 -> node-4.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.7 -> node-5.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.8 -> node-6.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.9 -> node-7.domain.tld
The pattern matches both the files in the symlink and in the actual directory producing duplicates. logrotate processes the first match in the symlinked dir and creates a log.1 file. When it comes around to the other match in the actual directory, there is already a .1 file which has not yet been compressed because of the processing in the symlinked dir. logrotate exits without completion with an error "file.1 exists". Further processing is aborted and files do not get rotated on subsequent runs. This eventually leads to a full file system.
A potential solution would be to have puppet write this pattern as /var/log/
Changed in fuel: | |
status: | New → Confirmed |
importance: | Undecided → High |
assignee: | nobody → Fuel Library Team (fuel-library) |
milestone: | none → 5.1 |
tags: | added: customer-found |
Changed in fuel: | |
status: | Fix Released → Fix Committed |
Changed in fuel: | |
status: | Fix Committed → Fix Released |
tags: | added: logrotate |
This was seen on fuel 4.1 but I believe it is an issue in later releases as well.