Mirantis OpenStack

Swift upstart mechanism is broken on controllers

Bug #1466101 reported by Ksenia Svechnikova on 2015-06-17

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Fix Released	Medium	MOS Packaging Team	Mirantis OpenStack 7.0
5.1.x	Fix Released	High	Alexander Nevenchannyy	Mirantis OpenStack 5.1.1-mu-2
6.0.x	Fix Released	High	Alexander Nevenchannyy	Mirantis OpenStack 6.0-mu-5
6.1.x	Fix Released	Medium	Alexander Nevenchannyy	Mirantis OpenStack 6.1-mu-3
7.0.x	Fix Released	Medium	MOS Packaging Team	Mirantis OpenStack 7.0

Bug Description

Version: 6.1, #521

Steps to reproduce:

1. Install environment with Swift (3 controllers and 2 compute nodes)

2. Check swift-object-replicator is running in swift-init:

root@node-4:~# swift-init all status | grep replicat
object-replicator running (8320 - /etc/swift/object-server.conf)
container-replicator running (8321 - /etc/swift/container-server.conf)
account-replicator running (8311 - /etc/swift/account-server.conf)

3. Check the process is running:

root@node-4:~# ps aux | grep swift-object-replicator
root 6809 0.0 0.0 10460 944 pts/3 S+ 12:59 0:00 grep --color=auto swift-object-replicator
swift 8320 0.2 0.0 1554688 26088 ? Ssl Jun16 3:08 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf

root@node-4:~# service swift-object-replicator status
swift-object-replicator start/running

4. Kill the process and confirm it is killed:

root@node-4:~# kill 8320
root@node-4:~# ps aux | grep swift-object-replicator
root 13225 0.0 0.0 10460 940 pts/3 S+ 13:02 0:00 grep --color=auto swift-object-replicator

Then you can see not appropriate behavior of upstart mechanism:

1. The "service swift-xxx start/stop/status" doesn't have valid information about the process status after the process is killed:

root@node-4:~# service swift-object-replicator status
swift-object-replicator start/running

root@node-4:~# ps aux | grep swift-object-replicator
root 13225 0.0 0.0 10460 940 pts/3 S+ 13:02 0:00 grep --color=auto swift-object-replicator

root@node-4:~# service swift-object-replicator start
start: Job is already running: swift-object-replicator

2. The pid file is still there, while it should be removed, when process is killed:

root@node-4:~# ps -ef | grep object-replicator
swift 11609 1 0 13:22 ? 00:00:01 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf
root 22261 24310 0 13:29 pts/3 00:00:00 grep --color=auto object-replicator
root@node-4:~# kill 11609

root@node-4:~# cat /var/run/swift/object-replicator.pid
11609

Meanwhile, if after the process is killed you will run "service stop", it will give a message "Job failed while stopping", but it will delete the pid and "service status" will work fine:

root@node-4:~# kill 11609

root@node4: service swift-object-replicator stop
stop: Job failed while stopping

root@node-4:~# service swift-object-replicator status
swift-object-replicator stop/waiting

root@node-4:~# cat /var/run/swift/object-replicator.pid
cat: /var/run/swift/object-replicator.pid: No such file or directory

See original description

Tags:

Ksenia Svechnikova (kdemina) on 2015-06-17

Changed in mos:
importance:	Undecided → Medium

Alexey Khivin (akhivin) on 2015-06-17

tags:

added: swift

Dmitry Mescheryakov (dmitrymex) on 2015-06-19

Changed in mos:
status:	New → Confirmed

Revision history for this message

Ivan Udovichenko (iudovichenko) wrote on 2015-06-19:

"2. The pid file is still there, while it should be removed, when process is killed"
Please check http://unixhelp.ed.ac.uk/CGI/man-cgi?signal+7

Could you please suggest how to catch and process SIGKILL?

Revision history for this message

Ksenia Svechnikova (kdemina) wrote on 2015-06-19:

Thanks for remark . I have updated the descpition, as it's reproduced with signal TERM

description:

updated

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-17: Related fix merged to openstack-build/swift-build (openstack-ci/fuel-7.0/2015.1.0)

Reviewed: https://review.fuel-infra.org/9494
Submitter: Igor Yozhikov <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 83ac1c99875e5e6c26c6686bc938f5862f7799d7
Author: Ivan Udovichenko <email address hidden>
Date: Fri Jul 17 12:14:48 2015

Update Swift Upstart scripts

Add respawn stanza to Swift .upstart scripts

Related-Bug: #1466101

Change-Id: I3158fb79f9350d410f253f8402b9ab33b4595afe

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-07-20:

Changing importance to High for 5.1/5.1.1-updates because it's customer found and assigning to Alexey Khivin for backporting.

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-24: Related fix proposed to openstack-build/swift-build (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Related fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/9849

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-24: Related fix merged to openstack-build/swift-build (openstack-ci/fuel-5.1.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/9849
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: 193ee4097e0e752b972a1b85bfb31fbe8c7d094a
Author: Alexandr Nevenchannyy <email address hidden>
Date: Fri Jul 24 14:08:35 2015

Update Swift Upstart scripts

Add respawn stanza to Swift .upstart scripts

Related-Bug: #1466101

Change-Id: I3158fb79f9350d410f253f8402b9ab33b4595afe

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-24: Related fix proposed to openstack-build/swift-build (openstack-ci/fuel-5.1-updates/2014.1.1)

Related fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/9861

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-24: Related fix merged to openstack-build/swift-build (openstack-ci/fuel-5.1-updates/2014.1.1)

Reviewed: https://review.fuel-infra.org/9861
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: 7806779982932d80e95c5fd9ccc9fca530d2c80c
Author: Alexandr Nevenchannyy <email address hidden>
Date: Fri Jul 24 16:20:34 2015

Update Swift Upstart scripts

Add respawn stanza to Swift .upstart scripts

Related-Bug: #1466101

Change-Id: I3158fb79f9350d410f253f8402b9ab33b4595afe
(cherry picked from commit 193ee4097e0e752b972a1b85bfb31fbe8c7d094a)

Revision history for this message

Sergey Novikov (snovikov) wrote on 2015-07-27:

I have reproduced a similar behavior after applying the fix on one controller. According to https://review.fuel-infra.org/9861 I edited the following files:

/etc/init/swift-proxy.conf
/etc/init/swift-account-replicator.conf
/etc/init/swift-container-auditor.conf
/etc/init/swift-container.conf
/etc/init/swift-object-replicator.conf
/etc/init/swift-container-updater.conf
/etc/init/swift-container-replicator.conf
/etc/init/swift-account.conf
/etc/init/swift-object.conf
/etc/init/swift-container-sync.conf
/etc/init/swift-account-auditor.conf
/etc/init/swift-account-reaper.conf
/etc/init/swift-object-updater.conf

Then I rebooted controller and executed steps to reproduce. Output is below:

root@node-5:~# swift-init all status | grep replicat
object-replicator running (8835 - /etc/swift/object-server.conf)
container-replicator running (4972 - /etc/swift/container-server.conf)
account-replicator running (4982 - /etc/swift/account-server.conf)

root@node-5:~# ps -ef | grep object-replicator
swift 8835 1 2 15:04 ? 00:00:00 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf
root 9034 45895 0 15:04 pts/0 00:00:00 grep --color=auto object-replicator

root@node-5:~# service swift-object-replicator status
swift-object-replicator start/running

root@node-5:~# kill 8835

root@node-5:~# swift-init all status | grep replicat
No object-replicator running
container-replicator running (4972 - /etc/swift/container-server.conf)
account-replicator running (4982 - /etc/swift/account-server.conf)

root@node-5:~# ps -ef | grep object-replicator
root 11602 45895 0 15:05 pts/0 00:00:00 grep --color=auto object-replicator

root@node-5:~# cat /var/run/swift/object-replicator.pid
cat: /var/run/swift/object-replicator.pid: No such file or directory

root@node-5:~# service swift-object-replicator status
swift-object-replicator start/running

ISO is custom, but it base on 5.1

Revision history for this message

Sergey Novikov (snovikov) wrote on 2015-07-29:

#10

Behavior which is described above can be reproduced on ISO:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1.2"
  api: "1.0"
  build_number: "354"
  build_id: "2015-07-28_08-59-18"
  astute_sha: "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce"
  fuellib_sha: "afed4badf133fe65543676ecf650209e38f7f892"
  ostf_sha: "9d4392fb97252c84dee7d29ee33c706d78b5bbef"
  nailgun_sha: "661a4f9502d2e6101ed2f5eb995bb675e6c3b2c5"
  fuelmain_sha: "14e6f6e4f23de077ac69f0f89dda40d2bea06536"

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-31: Fix proposed to openstack-build/swift-build (openstack-ci/fuel-5.1.1-updates/2014.1.1)

#11

Fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/10005

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-31: Fix merged to openstack-build/swift-build (openstack-ci/fuel-5.1.1-updates/2014.1.1)

#12

Reviewed: https://review.fuel-infra.org/10005
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: 3920a7e4ba3d95512a36c98df2cca260c707f78b
Author: Alexandr Nevenchannyy <email address hidden>
Date: Fri Jul 31 12:47:21 2015

Update Swift Upstart scripts

Changes are required in order to process correctly
process respawns during unexpected shutdown of it.
These changes are required because of Upstart
limitations when program makes more than 2
fork/clone operations [1]

[1] http://upstart.ubuntu.com/cookbook/#implications-of-misspecifying-expect

Closes-bug: #1466101
Change-Id: Id51f7c03cf4340e0022d53cff050511409db038d

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-31: Fix proposed to openstack-build/swift-build (openstack-ci/fuel-5.1-updates/2014.1.1)

#13

Fix proposed to branch: openstack-ci/fuel-5.1-updates/2014.1.1
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/10013

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-31: Fix merged to openstack-build/swift-build (openstack-ci/fuel-5.1-updates/2014.1.1)

#14

Reviewed: https://review.fuel-infra.org/10013
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: 1e8f055fb1f89adcc5e4337dc57a8d6415203d79
Author: Alexandr Nevenchannyy <email address hidden>
Date: Fri Jul 31 13:46:38 2015

Update Swift Upstart scripts

[1] http://upstart.ubuntu.com/cookbook/#implications-of-misspecifying-expect

Closes-bug: #1466101
Change-Id: Id51f7c03cf4340e0022d53cff050511409db038d
(cherry picked from commit 3920a7e4ba3d95512a36c98df2cca260c707f78b)

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-08-03: Related fix proposed to openstack-build/swift-build (openstack-ci/fuel-6.0-updates/2014.2)

#15

Related fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Ivan Udovichenko <email address hidden>
Review: https://review.fuel-infra.org/10039

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-08-03: Fix proposed to openstack-build/swift-build (openstack-ci/fuel-6.0-updates/2014.2)

#16

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Ivan Udovichenko <email address hidden>
Review: https://review.fuel-infra.org/10040

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-08-03: Related fix merged to openstack-build/swift-build (openstack-ci/fuel-6.0-updates/2014.2)

#17

Reviewed: https://review.fuel-infra.org/10039
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: b3815eb8ab8f3100df6a081b6f2edb516938cfa0
Author: Ivan Udovichenko <email address hidden>
Date: Mon Aug 3 13:44:17 2015

Update Swift Upstart scripts

Add respawn stanza to Swift .upstart scripts

Related-Bug: #1466101

Change-Id: I3158fb79f9350d410f253f8402b9ab33b4595afe

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-08-03: Fix merged to openstack-build/swift-build (openstack-ci/fuel-6.0-updates/2014.2)

#18

Reviewed: https://review.fuel-infra.org/10040
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: c77105b3d6b5b9729d5ccdd0074f2e8f15fea6bd
Author: Ivan Udovichenko <email address hidden>
Date: Mon Aug 3 13:44:17 2015

Update Swift Upstart scripts

[1] http://upstart.ubuntu.com/cookbook/#implications-of-misspecifying-expect

Closes-bug: #1466101
Change-Id: Id51f7c03cf4340e0022d53cff050511409db038d

Revision history for this message

Vadim Rovachev (vrovachev) wrote on 2015-08-06:

#19

Verified on 6.0

Olena Logvinova (ologvinova) on 2015-08-11

tags:

added: 6.0-mu-5 done release-notes

Olena Logvinova (ologvinova) on 2015-09-14

tags:

added: release-notes-done-7.0
removed: done release-notes

Olena Logvinova (ologvinova) on 2015-09-14

tags:

added: release-notes-done rn6.0-mu-5 rn7.0
removed: 6.0-mu-5 release-notes-done-7.0

Roman Rufanov (rrufanov) on 2015-09-17

tags:

added: support

Revision history for this message

Vitaly Sedelnik (vsedelnik) wrote on 2015-09-26:

#20

Nominated for 5.1x series, need to ensure the fix landed to appropriate branch

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-05: Related fix proposed to openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#21

Related fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/12492

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-05: Fix proposed to openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#22

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/12493

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-05: Change abandoned on openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#23

Change abandoned by Alexandr Nevenchannyy <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/12492
Reason: Required changes are too big to backport from kilo to juno release.

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-05: Change restored on openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#24

Change restored by Alexandr Nevenchannyy <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/12492
Reason: restore

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-06: Related fix merged to openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#25

Reviewed: https://review.fuel-infra.org/12492
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: ffac6ddbe7a655b22bd8fa7934c3566819f53bb0
Author: Alexandr Nevenchannyy <email address hidden>
Date: Mon Oct 5 12:07:54 2015

Update Swift Upstart scripts

Add respawn stanza to Swift .upstart scripts

Related-Bug: #1466101

Conflicts:
trusty/debian/changelog

Change-Id: I3158fb79f9350d410f253f8402b9ab33b4595afe

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-14: Fix merged to openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#26

Reviewed: https://review.fuel-infra.org/12493
Submitter: Artem Silenkov <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 20133631aa0fc2afa5960f8122cfc150af053dd7
Author: Alexandr Nevenchannyy <email address hidden>
Date: Mon Oct 5 12:17:31 2015

Update Swift Upstart scripts

[1] http://upstart.ubuntu.com/cookbook/#implications-of-misspecifying-expect

Closes-bug: #1466101

Conflicts:
debian/swift-account.swift-account-auditor.upstart
debian/swift-account.swift-account-reaper.upstart
debian/swift-account.swift-account-replicator.upstart
debian/swift-account.upstart
debian/swift-container.swift-container-auditor.upstart
debian/swift-container.swift-container-replicator.upstart
debian/swift-container.swift-container-sync.upstart
debian/swift-container.swift-container-updater.upstart
debian/swift-container.upstart
debian/swift-object-expirer.upstart
debian/swift-object.swift-object-auditor.upstart
debian/swift-object.swift-object-replicator.upstart
debian/swift-object.swift-object-updater.upstart
debian/swift-object.upstart
debian/swift-proxy.upstart

Change-Id: Id51f7c03cf4340e0022d53cff050511409db038d

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-10-15: Fix proposed to openstack-build/swift-build (openstack-ci/fuel-6.1/2014.2)

#27

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/12880