Periodic tasks don't use nova_compute_url setting

Bug #1289101 reported by Sam Morrison
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Alex Tomic

Bug Description

When periodic tasks run the calls to nova don't go to the endpoint specified in nova_compute_url

When taskmanager.Manager sets up a admin_context to use for periodic tasks it passes in nova_proxy_admin_tenant_name for the tenant attribute for the context. (A standard openstack context uses tenant_id for the tenant attribute, so this is confusing)

That admin context is used to create a nova client in common.remote
It creates the client correctly but then when it overrides the management_url it sets it to <nova_compute_url>/<nova_proxy_admin_tenant_name>

This is the wrong URL, it should be the tenant ID not the name. Nova seems to then ignore this value and in our case uses the default compute endpoint gotten from keystone.

Changed in trove:
importance: Undecided → Low
Changed in trove:
status: New → Triaged
Li Ma (nick-ma-z)
Changed in trove:
assignee: nobody → Li Ma (nick-ma-z)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/111679

Changed in trove:
status: Triaged → In Progress
Revision history for this message
Amrith Kumar (amrith) wrote :

I'm raising the priority of this bug as it is something that has hit several of us. It was discussed on IRC yesterday [approx 11:39 Eastern]. A fix is available and has been verified to work by multiple people.

Changed in trove:
importance: Low → High
milestone: none → kilo-1
Changed in trove:
milestone: kilo-1 → kilo-2
Revision history for this message
Sergey Gotliv (sgotliv) wrote :

Periodic task mentioned in that case is actually "publish_exist_events" - notification about active instances. These notifications may be very useful, especially if you want to charge a user based on the numbers of the active instances (thanks to slicknik for the great explanation), however those who experience the problem described in that case and doesn't need these notifications can turn it off by setting "exists_notification_transformer" configuration parameter in trove-taskmanager.conf to None.

exists_notification_transformer = None

Changed in trove:
milestone: kilo-2 → kilo-3
Li Ma (nick-ma-z)
Changed in trove:
assignee: Li Ma (nick-ma-z) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on trove (master)

Change abandoned by Li Ma (<email address hidden>) on branch: master
Review: https://review.openstack.org/111679
Reason: still not sure about every detail. it needs further investigation.

Alex Tomic (atomic777)
Changed in trove:
assignee: nobody → Alex Tomic (atomic777)
Revision history for this message
Alex Tomic (atomic777) wrote :

After spending some time reviewing the code and the proposed patches, it seems that there are three separate issues here.

1 - As described in the bug description, nova_proxy_admin_tenant_name is a tenant string as opposed to an id, and in nova_client() this name is appended to nova_compute_url incorrectly. The fix proposed by Li Ma to deprecate tenant_name in favour of tenant_id in this case makes sense.

2 - The need for bypass_url to fix the issue that Sam mentioned on a comment to patch set 10 feels like it should be addressed in a separate bug, because it could affect all code that creates connections to nova, not just admin connections that this bug addresses.

3 - Notifications will not work if nova_compute_url is not provided. The admin context is not created with a service_catalog, leading get_endpoint() to throw an EmptyCatalog exception. A nice-to-have would be to make notifications work without the need to hardcode a URL in a configuration file, or at least find a more graceful way of informing an operator that both parameters need to be set.

I propose that we keep this bug for the first issue above, and I will create two new bugs for the other two.

Revision history for this message
Alex Tomic (atomic777) wrote :

The two new bugs created are:

https://bugs.launchpad.net/trove/+bug/1431853 - Periodic tasks against nova can be incorrectly redirected after token expiry
https://bugs.launchpad.net/trove/+bug/1431848 - Exists notifications do not work if nova_compute_url is not provided

I have not reproduced the issue in the first bug, so comments/clarification from Sam or Li on whether i've described the issue correctly would be great.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/164224

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to trove (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/164232

Changed in trove:
milestone: kilo-3 → kilo-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to trove (master)

Reviewed: https://review.openstack.org/164232
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=daf4bb5847eb326c633756d369d2087899e6f8e8
Submitter: Jenkins
Branch: master

commit daf4bb5847eb326c633756d369d2087899e6f8e8
Author: Alex Tomic <email address hidden>
Date: Fri Mar 13 16:49:18 2015 +0100

    bypass_url required in nova admin client

    In the case of periodic tasks run in the task manager via the
    admin client, a bypass_url is needed to be passed on client
    connect in order to fix the endpoint should the token expire.
    Arose as part of investigation into bug 1289101 and associated
    review.

    Change-Id: I15e8a40e271d5b428eebfed6e99ec47a61611795
    Closes-Bug: #1431853
    Co-Authored-By: Li Ma <email address hidden>
    Related-Bug: #1289101

Changed in trove:
milestone: kilo-rc1 → liberty-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/164224
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=75088205f1e7b491c83f3638aef35995f1afab92
Submitter: Jenkins
Branch: master

commit 75088205f1e7b491c83f3638aef35995f1afab92
Author: Alex Tomic <email address hidden>
Date: Fri Mar 13 15:55:35 2015 +0100

    Change nova_proxy_admin_tenant_name to id

    The task manager can be configured with periodic events
    that check for the existence of running nova instances.
    The events run via a separate admin connection to nova.

    Taskmanager.Manager populates the admin client context
    with the tenant name provided by nova_proxy_admin_tenant_name
    parameter instead of the uuid, which results in an invalid
    management url that is composed of the two parameters:
    <nova_compute_url>/<nova_proxy_admin_tenant_name>.

    Changing tenant_name to tenant_id results in a valid endpoint.

    DocImpact
    deprecating option 'nova_proxy_admin_tenant_name'
    to 'nova_proxy_admin_tenant_id'

    Change-Id: Ia1315e41288ab1b24ac402bad15176cb1ae0e5cd
    Co-Authored-By: Li Ma <email address hidden>
    Closes-Bug: #1289101

Changed in trove:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: liberty-1 → 4.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.