tripleo

job that test ceph in periodic pipeline for stable branches are getting the wrong configuration

Bug #1817688 reported by Gabriele Cerami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Gabriele Cerami
tripleo stein-3

Bug Description

https://review.openstack.org/637196 Introduced a bug for the stable periodic jobs.

Affected jobs:

- periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky
- periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-rocky
- periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens
- periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens

An example at

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-rocky/4b9ccbc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-02-26_07_58_32

error:

ceph_disk.main.Error: Error: Device /dev/loop3 is in use by a device-mapper mapping (dm-crypt?): dm-0,dm-1,dm-2

Master jobs scenario001 and 004 are affected currently waiting for the migration patches to bluestore

As yatin commented in periodic jobs the branch is always master, and stable branches get a configuration that is not compatible with the branch.

The solution would be to pass the release variable, but it's not available at that stage in the playbook.

We need to make the release variable available earlier in the process.

See original description
Gabriele Cerami (gcerami)
description: updated
wes hayutin (weshayutin)
tags: removed: ci promotion-blocker quickstart
Gabriele Cerami (gcerami)
description: updated
wes hayutin (weshayutin)
tags: added: promotion-blocker
tags: added: ci
Revision history for this message
John Fulton (jfulton-org) wrote :

https://review.openstack.org/#/c/547682 can be backported to queens and rocky and they'll be green again. both use ceph-ansible 3.2 which supports bluestore.

pike EOLs in 5 days. i'll start backports and we can vote on this way forward or others.

https://releases.openstack.org/

Revision history for this message
wes hayutin (weshayutin) wrote :

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/75b3baa/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-02-26_07_41_45

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/639351

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/639352

Revision history for this message
John Fulton (jfulton-org) wrote :

Plan:
- master 001/004 are green and should vote; merge [0]
- confirm rocky/queens 001/004 turn green from [1][2]
- enable voting 001/004 in stable rocky/queens/pike
- stable will be voting again but pike jobs will be broken as it cannot use bluestore
- pike EOLs in five days [3] so a separate thread will cover disabling it

[0] https://review.openstack.org/#/c/638364
[1] https://review.openstack.org/#/c/639351
[2] https://review.openstack.org/#/c/639352
[3] https://releases.openstack.org

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/639351
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=61911edb5deafd2d28a09a540400fb524fd46ab9
Submitter: Zuul
Branch: stable/rocky

commit 61911edb5deafd2d28a09a540400fb524fd46ab9
Author: John Fulton <email address hidden>
Date: Fri Feb 23 17:10:08 2018 -0500

    Switch scenario00{1,4}-multinode-containers to Ceph bluestore

    Modify scenario00{1,4}-multinode to use Ceph's bluestore in place
    filestore. Bluestore is the default deployment method as of version
    3.2 of ceph-ansible so we should test it in CI. Use pre-created
    lvm_volumes parameter to avoid issue with 'ceph-volume batch'
    mode which does not work on loopback devices.

    blueprint: bluestore
    Depends-On: I747ac3dca5afdc91538da40b9ed45591ac8d1662
    Fixes-Bug: #1817688
    (cherry-picked from commit e3f697df6e1ff57b6f9e2cf3d455984dbf1faf84)

    Change-Id: Id2658ae814b580971d559af616b8ba034dff681b

tags: added: in-stable-rocky
tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/639352
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a89cd6b19b7bd4178e052aa4cf0ef7f59145b708
Submitter: Zuul
Branch: stable/queens

commit a89cd6b19b7bd4178e052aa4cf0ef7f59145b708
Author: John Fulton <email address hidden>
Date: Fri Feb 23 17:10:08 2018 -0500

    Switch scenario00{1,4}-multinode-containers to Ceph bluestore

    Modify scenario00{1,4}-multinode to use Ceph's bluestore in place
    filestore. Bluestore is the default deployment method as of version
    3.2 of ceph-ansible so we should test it in CI. Use pre-created
    lvm_volumes parameter to avoid issue with 'ceph-volume batch'
    mode which does not work on loopback devices.

    blueprint: bluestore
    Depends-On: I747ac3dca5afdc91538da40b9ed45591ac8d1662
    Fixes-Bug: #1817688
    (cherry-picked from commit e3f697df6e1ff57b6f9e2cf3d455984dbf1faf84)

    Change-Id: Id2658ae814b580971d559af616b8ba034dff681b

Revision history for this message
wes hayutin (weshayutin) wrote :

waiting to verify w/ queens and rocky jobs.

Changed in tripleo:
status: Triaged → Fix Released
status: Fix Released → Triaged
Revision history for this message
Gabriele Cerami (gcerami) wrote :

Rocky passes

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-rocky/e2e5412/

Queens runs only once a day, and last time it could not catch the latest tht version with the fix

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/784cf5c/logs/undercloud/var/log/extra/rpm-list.txt.gz

But I think we are good.

Changed in tripleo:
status: Triaged → In Progress
status: In Progress → Fix Committed
Revision history for this message
Gabriele Cerami (gcerami) wrote :

Queens passes too now:

https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/a12b745/job-output.txt.gz

We are good.

Changed in tripleo:
status: Fix Committed → Fix Released
Revision history for this message
John Fulton (jfulton-org) wrote :

Happening on tripleo-ci-centos-7-scenario010-multinode-oooq-container [1] e.g. [2] too. I will submit a follow up fix.

[1] http://logs.openstack.org/62/638762/2/check/tripleo-ci-centos-7-scenario010-multinode-oooq-container/13211f0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-03-04_22_43_48

[2] https://review.openstack.org/#/c/638762/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/641016

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.openstack.org/641016
Reason: https://review.openstack.org/#/c/641479 merged so multinode master will work with filestore. We can move it to bluestore in the future if we want

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.