Trove-taskmanager fails to start if urls obtained from Keystone

Bug #1402055 reported by Mark Biciunas
72
This bug affects 15 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
In Progress
Medium
Chhavi Agarwal
puppet-trove
Triaged
Undecided
Unassigned

Bug Description

If urls for nova / cinder / swift are not specified in the config files, trove is supposed to resolve them against Keystone. However, trove-taskmanager fails start start with the following error:

Traceback (most recent call last):
  File "/usr/bin/trove-taskmanager", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/trove/cmd/common.py", line 65, in run
    return main_function(conf)
  File "/usr/lib/python2.7/site-packages/trove/cmd/taskmanager.py", line 34, in main
    startup(conf, None)
  File "/usr/lib/python2.7/site-packages/trove/cmd/taskmanager.py", line 27, in startup
    topic=topic)
  File "/usr/lib/python2.7/site-packages/trove/common/rpc/service.py", line 36, in __init__
    self.manager_impl = importutils.import_object(manager)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/importutils.py", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 44, in __init__
    context=self.admin_context)
  File "/usr/lib/python2.7/site-packages/trove/openstack/common/importutils.py", line 38, in import_object
    return import_class(import_str)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/trove/extensions/mgmt/instances/models.py", line 234, in __init__
    self.nova_client = remote.create_admin_nova_client(self.context)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 117, in create_admin_nova_client
    client = create_nova_client(context)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 100, in nova_client
    endpoint_region=CONF.os_region_name)
  File "/usr/lib/python2.7/site-packages/trove/common/remote.py", line 56, in get_endpoint
    raise exception.EmptyCatalog()
EmptyCatalog: Empty catalog.

Supplying url information in the trove-taskmanager.conf file allows the service to start normaly.

Revision history for this message
Denis M. (dmakogon) wrote :

This doesn't seem to be a bug, since you have to supply correct OpenStack region.

Revision history for this message
Sushil Kumar (sushil-kumar2) wrote :

Agreed to Denis, there is the need of providing correct OpenStack region using os_region_name in the config files.

Changed in trove:
status: New → Invalid
Revision history for this message
Sushil Kumar (sushil-kumar2) wrote :

Also for a devstack setup with more than one region, this is gonna be fixed using https://review.openstack.org/#/c/140121/

whereas if you have any other region applicable you can specify the same in configs.

Revision history for this message
Mark Biciunas (mbiciunas) wrote :

Thank you for responding.

However, os_region_name was specified in trove-taskmanager.conf and is a valid region. So this can't be the issue.

Also, there is only one region defined = RegionOne.

The following is the relevant section from trove-taskmanager.conf:

    # Configuration options for talking to nova via the novaclient.
    trove_auth_url = http://10.40.10.230:5000/v2.0
    #nova_compute_url = http://localhost:8774/v2
    #nova_compute_url = http://10.40.10.230:8774/v2
    #cinder_url = http://localhost:8776/v1
    #cinder_url = http://10.40.10.230:8776/v1
    #swift_url = http://localhost:8080/v1/AUTH_
    #swift_url = http://10.40.10.230:8080/v1/AUTH_
    #neutron_url = http://localhost:9696/

    # nova_compute_url, cinder_url, swift_url, and heat_url can all be fetched
    # from Keystone. To fetch from Keystone, comment out nova_compute_url,
    # cinder_url, swift_url, and heat_url and optionally uncomment the lines below.

    # Region name of this node. Used when searching catalog. Default value is None.
    os_region_name = RegionOne
    # Service type to use when searching catalog.
    nova_compute_service_type = compute
    # Service type to use when searching catalog.
    cinder_service_type = volumev2
    # Service type to use when searching catalog.
    swift_service_type = object-store
    # Service type to use when searching catalog.
    #heat_service_type = orchestration
    # Service type to use when searching catalog.
    #neutron_service_type = network

Some notes about our configuration:
    - Network is nova-network, no neutron.
    - Distribution is RedHat Openstack 5 running on Red Hat Linux 7
    - Installed using packstack
    - All services are on one machine, save for Trove which has been installed to a separate machine.

Finally, as per ther orginal bug filing, Trove works if the urls are specified. So the issue is strictly with url lookup.

I think the status on this bug should be changed back to New and properly investigated. If there is further information I can give you to help reproduce, please let me know.

Changed in trove:
status: Invalid → New
Changed in trove:
importance: Undecided → Medium
assignee: nobody → Nikhil Manchanda (slicknik)
Revision history for this message
Javier Peña (jpena-c) wrote :

I was hit by the same issue. As a workaround, setting "exists_notification_transformer = " (empty value) in trove-taskmanager.conf worked for me.

I'm quite sure I got the solution somewhere online, but I cannot find it now, so leaving it here in case it helps someone else.

Changed in puppet-trove:
status: New → Triaged
Revision history for this message
Doug Shelley (0-doug) wrote :

The issue here is that the Task Manager sets up an exists notification transformer during it's startup sequence:

(from taskmanager/manager.py)
     self.admin_context = TroveContext(
            user=CONF.nova_proxy_admin_user,
            auth_token=CONF.nova_proxy_admin_pass,
            tenant=CONF.nova_proxy_admin_tenant_id)
        if CONF.exists_notification_transformer:
            self.exists_transformer = importutils.import_object(
                CONF.exists_notification_transformer,
                context=self.admin_context)

The admin_context hasn't gone through the process of authenticating so the service_catalog property isn't set. During the initialization of the NovaNotificationTransformer:
(from extensions/mgmt/instances/models.py)
class NovaNotificationTransformer(NotificationTransformer):
    def __init__(self, **kwargs):
        super(NovaNotificationTransformer, self).__init__(**kwargs)
        self.context = kwargs['context']
        self.nova_client = remote.create_admin_nova_client(self.context)
        self._flavor_cache = {}

If you don't set CONF.nova_compute_url, the call to create_admin_nova_client will attempt to use the context.service_catalog which in this case is never set.

In order for this code to work, I believe the Task Manager is going to have to authenticate the admin_context with Keystone to get the service_catalog.

(BTW, the workaround mentioned above works because this code path is not executed if exists_notification_transformer is None.)

Amrith Kumar (amrith)
Changed in trove:
status: New → Confirmed
Revision history for this message
Amrith Kumar (amrith) wrote :

Nikhil, any word on this?

Changed in trove:
milestone: none → newton-1
Amrith Kumar (amrith)
Changed in trove:
milestone: newton-1 → ongoing
Changed in trove:
assignee: Nikhil Manchanda (slicknik) → Shaik Apsar (sa709c)
status: Confirmed → In Progress
Revision history for this message
Shaik Apsar (sa709c) wrote :

I'm done with the changes for the review and please review the changes.

Now trove-taskmanager service is able to startup without nova_compute_url and we don't need to be disable exists_notification_transformer as workaround.

Since we are using nova admin credentials with valid trove_auth_url, 'trove.extensions.mgmt.instances.models.NovaNotificationTransformer' can make use of trove.common.single_tenant_remote module to get the nova admin client.

Revision history for this message
Chhavi Agarwal (chhagarw) wrote :

Can you share the review for this bug, so I can have a look, I am also facing the similar issue

Revision history for this message
Tomasz Nowak (tnowak) wrote :
Revision history for this message
Chhavi Agarwal (chhagarw) wrote :

Checked the proposed fix, but this is not fixing the actual issue for EmptyCatalogue, rather its skipping the service catalogue, and using the single_remote_tenant.
To actually fix this issue, we need to use the Keystone Session while creating the TroveContext, and pass the actual auth_token, this will initialize the service_catalogue for Trove.
Similar approach is been used in openstack-heat project
https://github.com/openstack/heat/blob/master/heat/engine/clients/os/nova.py#L81

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on trove (master)

Change abandoned by amrith (<email address hidden>) on branch: master
Review: https://review.openstack.org/385584
Reason: abandoning for inactivity

Changed in trove:
assignee: Shaik Apsar (sa709c) → Chhavi Agarwal (chhagarw)
Revision history for this message
Manoj Kumar (manojnkumar) wrote :

Chhavi, do you plan to drop the change you proposed?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.