Space leak in /var/log
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Triaged
|
High
|
Fuel Sustaining |
Bug Description
I have found that space leaks in /var/log
It is caused by nginx
# df -hl /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/
# lsof -c nginx | grep deleted
nginx 18644 root 6w REG 253,2 1431908352 21758179 /var/log/
nginx 18644 root 7w REG 253,2 315231125504 21762537 /var/log/
nginx 18644 root 8w REG 253,2 77824 21758464 /var/log/
nginx 18644 root 9w REG 253,2 115408896 21762538 /var/log/
nginx 18645 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/
nginx 18645 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/
nginx 18645 dockerroot 8w REG 253,2 77824 21758464 /var/log/
nginx 18645 dockerroot 9w REG 253,2 115408896 21762538 /var/log/
nginx 18646 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/
nginx 18646 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/
nginx 18646 dockerroot 8w REG 253,2 77824 21758464 /var/log/
nginx 18646 dockerroot 9w REG 253,2 115408896 21762538 /var/log/
nginx 18647 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/
nginx 18647 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/
nginx 18647 dockerroot 8w REG 253,2 77824 21758464 /var/log/
nginx 18647 dockerroot 9w REG 253,2 115408896 21762538 /var/log/
nginx 18648 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/
nginx 18648 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/
nginx 18648 dockerroot 8w REG 253,2 77824 21758464 /var/log/
nginx 18648 dockerroot 9w REG 253,2 115408896 21762538 /var/log/
# dockerctl stop nginx; sleep 120; dockerctl start nginx
# df -hl /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/
Description of the environment:
Fuel8
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
api: "1.0"
build_number: "570"
build_id: "570"
fuel-nailgun_sha: "558ca91a854cf2
python-
fuel-agent_sha: "658be72c4b42d3
fuel-
astute_sha: "b81577a5b7857c
fuel-library_sha: "c2a335b5b725f1
fuel-ostf_sha: "3bc76a63a9e7d1
fuel-mirror_sha: "fb45b80d7bee58
fuelmenu_sha: "78ffc73065a967
shotgun_sha: "63645dea384a37
network-
fuel-upgrade_sha: "616a7490ec7199
fuelmain_sha: "d605bcbabf3153
# rpm -qa fuel-docker-*
fuel-docker-
Changed in fuel: | |
assignee: | nobody → Bartłomiej Piotrowski (bpiotrowski) |
Quick debugging session shows that logrotate configuration on the master node doesn't reload nginx properly. This line is supposed to "reload" nginx:
/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
Obviously nginx.pid doesn't exist because nginx is running in the container. post_rotate section should be expanded with following line:
dockerctl shell nginx nginx -s reopen