tripleo

Ceph client.bootstrap-osd auth entry overwritten

Bug #1769722 reported by Keith Schincke on 2018-05-07

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Incomplete	Medium	Keith Schincke

Bug Description

tht and puppet-ceph improperly/unnecessarily attempts to deploy a client.bootstrap-osd auth entry for cephx enabled ceph clusters.

This results in two problems:
1) a race condition between puppet-ceph creating client.bootstrap-osd files on all nodes and ceph copying the keyring file between the monitor nodes.

2) tht and puppet-ceph changing the ceph generated secret of the keyring.

Here is a description of the race condition events:
  1) monitor-1 executes ceph-key-client.bootstrap-osd
     to create a new keyring on the local file system
  2) monitor-2 executes ceph-key-client.bootstrap-osd
     to create a new keyring on the local file system
  3) monitor-2 executes ceph-inject-key-client.bootstrap-osd
     to add the client key and caps to the ceph auth list
  4) ceph will distribute the client.bootstrap-osd keyring to
     the other monitors. This will overwrite the keyring written
     in #1 with just the client name and cephx secret.
  5) monitor-1 executes ceph-injectkey-client.bootstrap-osd. The
     diff statement in the unless check will fail because the two
     keyrings will not return 0 because the one in $keyring_path will
     not contain any caps. The exec will fail with an "Error EINVAL:
     auth import: no caps supplied" error because the keyring file
     only has the client name and the cephx secret.

Steps to reproduce:
Both are timing dependent and may require multiple re-deployments to cause an expectant outcome.
The race condition will fail with the Error EINVAL: auth import: no caps supplied" error. The nature of the race condition makes it difficult to reproduce.

The secret overwrite will be seen when the diff in the unless statement of ceph::key::ceph-injectkey-${name} fails due to the secrets changing. This is a symptom of the design problem.

This fix will need to be backported to Ocata.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-07: Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/566701

Changed in tripleo:
status:	New → In Progress

Emilien Macchi (emilienm) on 2018-05-07

Changed in tripleo:
importance:	Undecided → Medium
milestone:	none → rocky-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-18: Fix proposed to tripleo-heat-templates (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/569376

Emilien Macchi (emilienm) on 2018-06-05

Changed in tripleo:
milestone:	rocky-2 → rocky-3

Emilien Macchi (emilienm) on 2018-07-26

Changed in tripleo:
milestone:	rocky-3 → rocky-rc1

Emilien Macchi (emilienm) on 2018-07-26

Changed in tripleo:
milestone:	rocky-rc1 → stein-1

Juan Antonio Osorio Robles (juan-osorio-robles) on 2018-10-30

Changed in tripleo:
milestone:	stein-1 → stein-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-12-29: Change abandoned on tripleo-heat-templates (stable/ocata)

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/569376
Reason: This patch has no activity for a while, per policy, we abandon it. Feel free to re-open if you think you're working on it.

Emilien Macchi (emilienm) on 2019-01-13

Changed in tripleo:
milestone:	stein-2 → stein-3

Revision history for this message

Juan Antonio Osorio Robles (juan-osorio-robles) wrote on 2019-03-05:

Is this still an issue?

Alex Schultz (alex-schultz) on 2019-03-14

Changed in tripleo:
milestone:	stein-3 → train-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-04: Change abandoned on tripleo-heat-templates (master)

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.opendev.org/566701
Reason: Abandoning 8-month old patch

Alex Schultz (alex-schultz) on 2019-06-07

Changed in tripleo:
milestone:	train-1 → train-2

Alex Schultz (alex-schultz) on 2019-07-29

Changed in tripleo:
milestone:	train-2 → train-3

Alex Schultz (alex-schultz) on 2019-09-11

Changed in tripleo:
milestone:	train-3 → ussuri-1

Emilien Macchi (emilienm) on 2019-12-19

Changed in tripleo:
milestone:	ussuri-1 → ussuri-2

wes hayutin (weshayutin) on 2020-02-10

Changed in tripleo:
milestone:	ussuri-2 → ussuri-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-11: Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/712348

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-13: Fix proposed to tripleo-heat-templates (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/712943

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-13: Change abandoned on tripleo-heat-templates (stable/pike)

Change abandoned by Giulio Fidente (<email address hidden>) on branch: stable/pike
Review: https://review.opendev.org/712943

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-14: Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/712348
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=440aa8d210163125cf0e72c7a23d4b4ca12cd72d
Submitter: Zuul
Branch: stable/queens

commit 440aa8d210163125cf0e72c7a23d4b4ca12cd72d
Author: Keith Schincke <email address hidden>
Date: Mon May 7 14:48:29 2018 -0400

Unnessary deploying of bootstrap-osd ceph auth entry

    Ceph jewel creates the client.bootstrap-osd auth entry when the cluster
    is created. heat does not need to pass this key to ceph::key as the
    auth will already be populated on the monitors and the OSD nodes will
    extract a copy from memory. This patch removes client.bootstrap-osd from
    the ceph::profile::params::client_keys array.

    Change-Id: Ia9c50afd3a88b27e6e0dca2ce150f91633681466
    Closes-bug: 1769722
    Depends-on: I11f134bed59da706a03ce4a70d941145c16f8175
    Depends-On: https://review.opendev.org/#/c/712501/

tags:

added: in-stable-queens

wes hayutin (weshayutin) on 2020-04-13

Changed in tripleo:
milestone:	ussuri-3 → ussuri-rc3

wes hayutin (weshayutin) on 2020-05-26

Changed in tripleo:
milestone:	ussuri-rc3 → victoria-1

Emilien Macchi (emilienm) on 2020-07-28

Changed in tripleo:
milestone:	victoria-1 → victoria-3

Marios Andreou (marios-b) on 2020-11-03

Changed in tripleo:
milestone:	victoria-3 → wallaby-1

Marios Andreou (marios-b) on 2020-12-08

Changed in tripleo:
milestone:	wallaby-1 → wallaby-2

Marios Andreou (marios-b) on 2021-01-29

Changed in tripleo:
milestone:	wallaby-2 → wallaby-3

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-02-19:

#10

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone:	wallaby-3 → none
status:	In Progress → Incomplete

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-07-12: Fix included in openstack/tripleo-heat-templates queens-eol

#11

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.