[SRU] glance-simplestreams-sync charm doesn't support keystone v3

Bug #1611987 reported by Brad Marshall
64
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Glance - Simplestreams Sync Charm
Invalid
Undecided
Unassigned
OpenStack Glance-Simplestreams-Sync Charm
Fix Released
High
Ryan Beisner
simplestreams
Fix Released
Medium
Unassigned
glance-simplestreams-sync (Juju Charms Collection)
Invalid
High
Unassigned
simplestreams (Ubuntu)
Fix Released
Medium
Unassigned
Xenial
Won't Fix
Medium
Unassigned
Artful
Won't Fix
Medium
Unassigned
Bionic
Fix Released
Medium
Unassigned

Bug Description

[Impact]

simplestreams can't sync images when keystone is configured to use v3, keystone v2 is deprecated since mitaka[0] (the version shipped with xenial)

The OpenStack Keystone charm supports v3 only since Queens and later[1]

[Test Case]

* deploy a openstack environment with keystone v3 enabled
  - get a copy of the bundle available at http://paste.ubuntu.com/p/hkhsHKqt4h/ , this bundle deploys a minimal version of xenial-mitaka.

Expected result:

- "glance image-list" lists trusty and xenial images
- the file glance-simplestreams-sync/0:/var/log/glance-simplestreams-sync.log contains details of the images pulled from cloud-images.u.c (example: https://pastebin.ubuntu.com/p/RWG8QrkVDz/ )

Actual result:

- "glance image-list" is empty
- the file glance-simplestreams-sync/0:/var/log/glance-simplestreams-sync.log contains the following stacktrace
INFO * 04-09 22:04:06 [PID:14571] * root * Calling DryRun mirror to get item list
ERROR * 04-09 22:04:06 [PID:14571] * root * Exception during syncing:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 471, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 232, in do_sync
    objectstore=store)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 374, in __init__
    super(ItemInfoDryRunMirror, self).__init__(config, objectstore)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 126, in __init__
    self.keystone_creds = openstack.load_keystone_creds()
  File "/usr/lib/python2.7/dist-packages/simplestreams/openstack.py", line 61, in load_keystone_creds
    raise ValueError("(tenant_id or tenant_name)")
ValueError: (tenant_id or tenant_name)

[Regression Potential]

* A possible regression will manifest itself figuring out if v2 or v3 should be used, after the connection is made there are no further changes introduced by this SRU

[Other Info]

I was deploying a Mitaka Trusty 16.04 charm based Openstack cloud (using the cloud archives), including glance-simplestreams-sync, using keystone v3.

Once I had everything deployed, the glance-simplestreams-sync service couldn't authenticate because it's using keystone v2, not v3, as you can see from the following:

INFO * 08-10 23:16:01 [PID:33554] * root * glance-simplestreams-sync started.
DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.session * REQ: curl -i -X POST http://x.y.z.240:5000/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"passwordCredentials": {"username": "image-stream", "password": "thisisnotapassword"}, "tenantId": "blahblahtenantidblahblah"}}'
INFO * 08-10 23:16:01 [PID:33554] * urllib3.connectionpool * Starting new HTTP connection (1): x.y.z.240
DEBUG * 08-10 23:16:01 [PID:33554] * urllib3.connectionpool * Setting read timeout to None
DEBUG * 08-10 23:16:01 [PID:33554] * urllib3.connectionpool * "POST /v2.0/tokens HTTP/1.1" 401 114
DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.session * RESP: [401] CaseInsensitiveDict({'content-length': '114', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.7 (Ubuntu)', 'date': 'Wed, 10 Aug 2016 23:16:01 GMT', 'www-authenticate': 'Keystone uri="http://x.y.z.240:5000"', 'x-openstack-request-id': 'req-f8aaf12d-01ea-46be-869a-6948ab38361b', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.session * Request returned failure status: 401
DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.v2_0.client * Authorization Failed.

Please update the charm to understand when its using keystone v3 and use the right auth methods.

Related bugs:
 * bug 1686437: glance sync: need keystone v3 auth support

Related branches

James Page (james-page)
Changed in glance-simplestreams-sync (Juju Charms Collection):
status: New → Triaged
milestone: none → 16.10
importance: Undecided → High
Liam Young (gnuoy)
Changed in glance-simplestreams-sync (Juju Charms Collection):
assignee: nobody → Liam Young (gnuoy)
Revision history for this message
Liam Young (gnuoy) wrote :

So this is fun.

Swift proxy v3 integration appears to be broken: Bug #1624304
The simplestreams package does not support v3 integration: Bug #1624306

Revision history for this message
Liam Young (gnuoy) wrote :
James Page (james-page)
Changed in glance-simplestreams-sync (Juju Charms Collection):
milestone: 16.10 → 17.01
tags: added: ks-v3 openstack
James Page (james-page)
Changed in charm-glance-simplestreams-sync:
assignee: nobody → Liam Young (gnuoy)
importance: Undecided → High
status: New → Triaged
Changed in glance-simplestreams-sync (Juju Charms Collection):
status: Triaged → Invalid
Changed in charm-glance-simplestreams-sync:
assignee: Liam Young (gnuoy) → Alex Kavanagh (ajkavanagh)
Felipe Reyes (freyes)
Changed in glance-simplestreams-sync (Juju Charms Collection):
assignee: Liam Young (gnuoy) → nobody
Changed in charm-glance-simplestreams-sync:
assignee: Alex Kavanagh (ajkavanagh) → nobody
Felipe Reyes (freyes)
Changed in glance-simplestreams-sync-charm:
status: New → Invalid
Nobuto Murata (nobuto)
tags: added: cpe-onsite
Revision history for this message
Junien Fridrick (axino) wrote :

Hi,

What's the status on this ?

This is absolutely required on queens, which dropped keystone v2 support.

Also, https://jujucharms.com/glance-simplestreams-sync/11 "submit a bug" links to https://bugs.launchpad.net/charms/+source/glance-simplestreams-sync/+filebug, should it link to https://bugs.launchpad.net/charm-glance-simplestreams-sync/+filebug instead ?

Thanks

Revision history for this message
Tom Haddon (mthaddon) wrote :

Subscribed ~field-high. This bug affects any cloud running Queens, which dropped keystone v2 support.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

This charm has no Queens integration tests, and it has no existing functional test suite to confirm upstream KSV3 commits, KSV2, or anything else for that matter.

I'm adding a basic functional test, based on the official glance charm, and augmenting that so that we can quickly exercise and assess this charm for the following:

 - Existing combos and older releases (@ KSV2)
 - Queens and later (KSV3)

To be clear, this effort does not add an automated test suite to the charm, but it does declare a reproducible test environment in line with other openstack charms. That test environment can be used to consistently develop and verify new and existing code.

It should be noted, also, that this charm lacks support for juju workload status. That means that this charm will not affirm a ready state for those waiting for a ready state through tools like juju-wait, leading to anticipated race conditions and certain automation blocking conditions. It is also not on the roadmap to provide that functionality in this charm.

Ultimately, the official glance openstack charm is slated to receive an image sync feature, and this GSS charm should be deprecated in favor of that when it is available.

We'll post our outcome of exercising the charm's git master against KSV3 combos as soon as possible.

Thank you.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

I'm finding that the gss charm at master does not result in images being sync'd, and I believe this is why:

https://bugs.launchpad.net/simplestreams/+bug/1686437

Scott Moser (smoser)
description: updated
Revision history for this message
David Ames (thedac) wrote :

The git version of glance-simplestreams-sync [0] is ready for keystone v3, however, we are blocked on the released version of simplestreams on xenial in Bug#1686437.

As a work around and to see the gsss charm work, on the gsss node run the following:

sudo apt-get install bzr
bzr branch lp:simplestreams
cd simplestreams
sudo python setup.py install

After this the sync will succeed.

We could discuss the possibility of putting simplestreams into the cloud archive and setting the gsss source accordingly.

[0] https://github.com/openstack-charmers/charm-glance-simplestreams-sync/
[1] https://bugs.launchpad.net/simplestreams/+bug/1686437

Revision history for this message
Scott Moser (smoser) wrote :

Hi,

Can someone test the PPA upload I've done at
 https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

apt-add-repository -y ppa:smoser/sstream-ks3

Revision history for this message
Ryan Beisner (1chb1n) wrote :

@smoser, i'll exercise that now. Thank you.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

@smoser, feedback on that posted to https://bugs.launchpad.net/simplestreams/+bug/1686437

Revision history for this message
David Ames (thedac) wrote :

@smoser

When using ppa:smoser/sstream-ks3 version 0.1.0~bzr426-0ubuntu1.3~ppa0

And a quick fix:
https://code.launchpad.net/~thedac/simplestreams/lp1611987/+merge/342971

I get an error:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 473, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 246, in do_sync
    tmirror.sync(smirror, path=initial_path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 346, in sync_products
    (prodname, vername))
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 368, in insert_version
    self._insert_item(*iargs)

Sure enough _insert_item does not exist in this version although it does exist in tip of lp:simplestreams.

Scott Moser (smoser)
Changed in simplestreams (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Changed in simplestreams:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → David Ames (thedac)
Changed in simplestreams (Ubuntu Xenial):
status: New → Confirmed
Changed in simplestreams (Ubuntu Artful):
status: New → Confirmed
Changed in simplestreams (Ubuntu Xenial):
importance: Undecided → Medium
Changed in simplestreams (Ubuntu Artful):
importance: Undecided → Medium
Ryan Beisner (1chb1n)
Changed in charm-glance-simplestreams-sync:
assignee: nobody → Ryan Beisner (1chb1n)
milestone: none → 18.05
Ryan Beisner (1chb1n)
Changed in charm-glance-simplestreams-sync:
status: Triaged → In Progress
Revision history for this message
Ryan Beisner (1chb1n) wrote :

As of today, the charm @ git master can deploy Xenial-Queens, using ksv3, and sync images, if the dev ppa of simplestreams is used. There are SRUs of simplestreams packages underway.

We've added a test suite to the charm as well, which is what has allowed us to confirm the combos which are ok, and raise simplestreams regression reports against those that are not. We will be adding that test suite to a charm test gate in gerrit ASAP, to keep it all healthy going forward.

The updated charm will not be released to the charm store until we can verify that it actually works where expected and that it does not regress functionality on the existing releases. This validation is in tandem with the simplestreams SRU.

#### Workaround and Verification:
Use the charm at current master and set the 'source' charm config at deploy-time to 'ppa:simplestreams-dev/trunk' to pick up the version of simplestreams which supports ksv3.

We have not yet tested a charm-upgrade and post-deploy package upgrade scenario, but that should also be possible. If anyone validates that before we do, please report back on this bug with your findings.

#### Repo:
https://github.com/openstack-charmers/charm-glance-simplestreams-sync

https://github.com/openstack-charmers/charm-glance-simplestreams-sync/commit/d8355af37c1b8683b33b39c873a672abdfa3eb2c

#### Workload status correction:
Earlier in this bug, I observed that this charm had not implemented workload status updates, but it has, with some caveats.

The charm will show in an unknown state until the first image sync begins. If the sync never begins, the workload status is never updated. That condition exists today by default with the released version of simplestreams on Xenial, when used with ksv3, hence no workload status in my initial test case.

Ex. when things go well, workload status is updated and helpful:

glance-simplestreams-sync/1* maintenance idle 6 10.5.0.3 Synchronising images

We will keep this bug updated as it all progresses.

Changed in charm-glance-simplestreams-sync:
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package simplestreams - 0.1.0~bzr460-0ubuntu1

---------------
simplestreams (0.1.0~bzr460-0ubuntu1) bionic; urgency=medium

  * New upstream snapshot.
    - Glance: handle Keystone v2 session based authentication.
      [David Ames] (LP: #1611987)

 -- Scott Moser <email address hidden> Thu, 12 Apr 2018 12:50:58 -0400

Changed in simplestreams (Ubuntu Bionic):
status: Confirmed → Fix Released
Revision history for this message
Junien Fridrick (axino) wrote :

I can confirm that the workaround proposed in #13 works. Small twist though : updating the "source" of an already installed glance-simplestream-sync application won't do anything, since the "source" option is only used at install time. You'll want to manually add the PPA and upgrade the python-simplestreams package.

Felipe Reyes (freyes)
description: updated
summary: - glance-simplestreams-sync charm doesn't support keystone v3
+ [SRU] glance-simplestreams-sync charm doesn't support keystone v3
tags: added: sts
Revision history for this message
Chris Gregan (cgregan) wrote :

Field High SLA now requires that a estimated date for a fix is listed in the comments. Please provide this estimate for the open tasks.

David Ames (thedac)
Changed in charm-glance-simplestreams-sync:
status: Fix Committed → Fix Released
Revision history for this message
Alvaro Uria (aluria) wrote :

When could we expect a simplestreams backport to xenial? Thank you.

Revision history for this message
Scott Moser (smoser) wrote :

Hi,
There is more information on a SRU of simplestreams in bug 1686437.

https://bugs.launchpad.net/simplestreams/+bug/1686437/comments/11

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Marking the simplestreams bug as fix released as keystone v3 was added to it in 2017-09-14 with this commit: https://git.launchpad.net/simplestreams/commit/?id=1dab951f16541ee29d72a80296e4b4f413761067

Changed in simplestreams:
status: Confirmed → Fix Released
assignee: David Ames (thedac) → nobody
Joshua Powers (powersj)
Changed in simplestreams (Ubuntu Artful):
status: Confirmed → Won't Fix
Changed in simplestreams (Ubuntu Xenial):
assignee: nobody → Joshua Powers (powersj)
assignee: Joshua Powers (powersj) → Rafael David Tinoco (rafaeldtinoco)
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :
Changed in simplestreams (Ubuntu Xenial):
status: Confirmed → In Progress
Revision history for this message
Robie Basak (racb) wrote :

> simplestreams can't sync images when keystone is configured to use v3, keystone v2 is deprecated since mitaka[0] (the version shipped with xenial)

I don't think this feature addition qualifies for an SRU under our usual policies. If an exception is needed we can discuss that, but we concluded on IRC in #ubuntu-devel that the expected way to fix this is via the Cloud Archive, rather than an SRU, so I'm rejecting this from the SRU upload queue.

Revision history for this message
Robie Basak (racb) wrote : Proposed package upload rejected

An upload of simplestreams to xenial-proposed has been rejected from the upload queue for the following reason: "See https://bugs.launchpad.net/charms/+source/glance-simplestreams-sync/+bug/1611987/comments/21".

Changed in simplestreams (Ubuntu Xenial):
status: In Progress → Won't Fix
assignee: Rafael David Tinoco (rafaeldtinoco) → nobody
Changed in simplestreams (Ubuntu Bionic):
assignee: nobody → Rafael David Tinoco (rafaeldtinoco)
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

TL;DR version:

We are going to fix Bionic simplestreams package and provide this Bionic simplestreams package in Ubuntu Cloud Archive for Xenial. With that, we are fixing Xenial simplestreams behavior (to work with keystone v3) only if end-user enables Ubuntu Cloud Archive.

Conversation about this topic:

https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/simplestreams/+git/simplestreams/+merge/373030/comments/980111

Changed in simplestreams (Ubuntu Bionic):
assignee: Rafael David Tinoco (rafaeldtinoco) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.