Space leak in /var/log

Bug #1648467 reported by Serg Lystopad
This bug report is a duplicate of:  Bug #1642794: nginx keeps writing to rotated logs. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Triaged
High
Fuel Sustaining

Bug Description

I have found that space leaks in /var/log
It is caused by nginx

# df -hl /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-varlog 332G 319G 0 100% /var/log

# lsof -c nginx | grep deleted
nginx 18644 root 6w REG 253,2 1431908352 21758179 /var/log/nginx/access_nailgun.log-20160825 (deleted)
nginx 18644 root 7w REG 253,2 315231125504 21762537 /var/log/nginx/error_nailgun.log-20160825 (deleted)
nginx 18644 root 8w REG 253,2 77824 21758464 /var/log/nginx/access_repo.log-20160825 (deleted)
nginx 18644 root 9w REG 253,2 115408896 21762538 /var/log/nginx/error_repo.log-20160825 (deleted)
nginx 18645 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/nginx/access_nailgun.log-20160825 (deleted)
nginx 18645 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/nginx/error_nailgun.log-20160825 (deleted)
nginx 18645 dockerroot 8w REG 253,2 77824 21758464 /var/log/nginx/access_repo.log-20160825 (deleted)
nginx 18645 dockerroot 9w REG 253,2 115408896 21762538 /var/log/nginx/error_repo.log-20160825 (deleted)
nginx 18646 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/nginx/access_nailgun.log-20160825 (deleted)
nginx 18646 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/nginx/error_nailgun.log-20160825 (deleted)
nginx 18646 dockerroot 8w REG 253,2 77824 21758464 /var/log/nginx/access_repo.log-20160825 (deleted)
nginx 18646 dockerroot 9w REG 253,2 115408896 21762538 /var/log/nginx/error_repo.log-20160825 (deleted)
nginx 18647 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/nginx/access_nailgun.log-20160825 (deleted)
nginx 18647 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/nginx/error_nailgun.log-20160825 (deleted)
nginx 18647 dockerroot 8w REG 253,2 77824 21758464 /var/log/nginx/access_repo.log-20160825 (deleted)
nginx 18647 dockerroot 9w REG 253,2 115408896 21762538 /var/log/nginx/error_repo.log-20160825 (deleted)
nginx 18648 dockerroot 6w REG 253,2 1431908352 21758179 /var/log/nginx/access_nailgun.log-20160825 (deleted)
nginx 18648 dockerroot 7w REG 253,2 315231125504 21762537 /var/log/nginx/error_nailgun.log-20160825 (deleted)
nginx 18648 dockerroot 8w REG 253,2 77824 21758464 /var/log/nginx/access_repo.log-20160825 (deleted)
nginx 18648 dockerroot 9w REG 253,2 115408896 21762538 /var/log/nginx/error_repo.log-20160825 (deleted)

# dockerctl stop nginx; sleep 120; dockerctl start nginx

# df -hl /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/os-varlog 332G 24G 292G 8% /var/log

Description of the environment:
Fuel8
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

# rpm -qa fuel-docker-*
fuel-docker-images-8.0.0-11.x86_64

Changed in fuel:
assignee: nobody → Bartłomiej Piotrowski (bpiotrowski)
Revision history for this message
Bartłomiej Piotrowski (bpiotrowski) wrote :

Quick debugging session shows that logrotate configuration on the master node doesn't reload nginx properly. This line is supposed to "reload" nginx:

/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true

Obviously nginx.pid doesn't exist because nginx is running in the container. post_rotate section should be expanded with following line:

dockerctl shell nginx nginx -s reopen

Changed in fuel:
importance: Undecided → High
status: New → Triaged
milestone: none → 8.0-mu-4
assignee: Bartłomiej Piotrowski (bpiotrowski) → Fuel Sustaining (fuel-sustaining-team)
Revision history for this message
Bartłomiej Piotrowski (bpiotrowski) wrote :

Reassigning to Fuel Sustaining as I have no time to go through formal review with my patch. Please see the first comment for the fix.

Revision history for this message
Serg Lystopad (slystopad) wrote :

Nether
cat /etc/logrotate.d/nginx
/var/log/nginx/*log {
    create 0644 nginx nginx
    daily
    rotate 10
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        #/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
        /usr/bin/dockerctl shell nginx nginx -s reopen
    endscript
}

nor

# dockerctl shell nginx
# cat /etc/logrotate.d/nginx
/var/log/nginx/*log {
    create 0644 nginx nginx
    daily
    rotate 10
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        #/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
        /usr/sbin/nginx -s reopen
    endscript
}

helps

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.