nova.task_log database records are never deleted

Bug #1877189 reported by melanie witt
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Wishlist
melanie witt
Train
New
Undecided
Unassigned
Ussuri
New
Undecided
Unassigned
Victoria
New
Undecided
Unassigned
Wallaby
In Progress
Undecided
Unassigned

Bug Description

nova.task_log database records are related to the server usage audit log API [1] which is dependent on the [DEFAULT]/instance_usage_audit config option being set to True (it defaults to False). Enablement of the config option causes nova-compute to emit notifications which are consumed by OpenStack Telemetry. When the config option is enabled, records will be periodically created in the nova.task_log table. And there is no cleanup of the records in nova.

I started a ML thread about the server usage audit log API awhile back [2] and the conclusion was that deprecating or removing the API isn't practical as there are systems in the wild still using it, so the best action to take for now is to create a nova-manage command for cleaning up old nova.task_log records.

I brought up the proposal in a nova meeting [3] and in the nova channel [4] and the consensus was to treat it as a Wishlist bug so that it can be backported upstream.

So this is a bug for the work to add a new 'nova-manage db purge_task_log' command that offers a --before <datetime> option.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-09-06.log.html#t2019-09-06T14:19:50
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009245.html
[3] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log.html#l-306
[4] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-10-17.log.html#t2019-10-17T14:54:39

Tags: nova-manage
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Wishlist
status: New → Confirmed
Revision history for this message
melanie witt (melwitt) wrote :
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/780395
Committed: https://opendev.org/openstack/nova/commit/bc71331bcaa92718fce06ac3d2ecda6a4c9d2d20
Submitter: "Zuul (22348)"
Branch: master

commit bc71331bcaa92718fce06ac3d2ecda6a4c9d2d20
Author: melanie witt <email address hidden>
Date: Thu Mar 11 23:47:05 2021 +0000

    Add --task-log option to nova-manage db archive_deleted_rows

    The task_log table contains instance usage audit records if
    nova-compute has been configured with [DEFAULT]instance_usage_audit =
    True. This will be the case if OpenStack Telemetry is being used in the
    deployment, as the option causes nova to generate audit information
    that Telemetry then retrieves from the server usage audit log API [1].

    Historically, there has been no way to delete task_log table records
    other than manual database modification. Because of this, task_log
    records could pile up over time and operators are forced to perform
    manual steps to periodically truncate the table.

    This adds a --task-log option to the 'nova-manage db
    archive_deleted_rows' CLI to also archive task_log records while
    archiving the database. --task-log works in conjunction with --before
    if operators desire archving only records that are older than <date>.
    The 'updated_at' field is used by --task-log --before <date> to
    determine the age of a task_log record for archival.

    Closes-Bug: #1877189

    [1] https://docs.openstack.org/api-ref/compute/#server-usage-audit-log-os-instance-usage-audit-log

    Change-Id: Ibed67854a693c930effd4dba7aca6cd03b65bd92

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/801144

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/801144
Committed: https://opendev.org/openstack/nova/commit/d093849c3c70e46bf409b1051454cbf70bc12a1f
Submitter: "Zuul (22348)"
Branch: master

commit d093849c3c70e46bf409b1051454cbf70bc12a1f
Author: melanie witt <email address hidden>
Date: Fri Jul 16 21:53:38 2021 +0000

    Correct reno for archiving task_log database records

    The release note incorrectly stated that the Telemetry service calls
    the /os-instance_usage_audit_log REST API to retrieve instance usage
    data. Telemetry rather consumes the notifications that are only generated
    when [DEFAULT]instance_usage_audit=True.

    Related-Bug: #1877189

    Change-Id: Iea358090c42877bedfdcc4a04019f6dc406176c2

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/802766

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.