simple tenant usage api calculating disk usages incorrectly

Bug #1715570 reported by Bhagyashri Shewale
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Medium
Shilpa Devharakar

Bug Description

As of now on current master the simple tenant usage api is calculating disk usages information incorrectly in case in instance is booted from volume.

If user boots the instance from volume then the disk usages should not be considered but it's taking the flavor.root_gb as local_gb used.

Steps to reproduce:

1. Create bootable volume

$ cinder create 1 --name bootable_volume --image 33a7f2aa-5e48-4545-a484-b598abdee1e3

+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+
| 0399f203-5f36-4b9a-b76f-bada8b9afdc6 | available | bootable_volume | 1 | lvmdriver-1 | true | |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+

2. Flavor details:

$ nova flavor-show 1
+----------------------------+---------+
| Property | Value |
+----------------------------+---------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 1 |
| extra_specs | {} |
| id | 1 |
| name | m1.tiny |
| os-flavor-access:is_public | True |
| ram | 512 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+---------+

3. Create instance from volume:

$ nova boot --flavor 1 --boot-volume 0399f203-5f36-4b9a-b76f-bada8b9afdc6 boot-from-volume

+--------------------------------------+------------------+--------+------------+-------------+---------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+--------+------------+-------------+---------------------------------+
| c46243b7-9034-4847-a9c5-480f5e60f59f | boot-from-volume | ACTIVE | - | Running | public=172.24.4.11, 2001:db8::8 |
+--------------------------------------+------------------+--------+------------+-------------+---------------------------------+

4. Check the simple tenant usage:

$ nova usage --tenant f77f8f0f6f1b46fe8a43b27a5372f427 --start 2017-09-07

Usage from 2017-09-07 to 2017-09-08:
+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 2 | 3602.99 | 7.04 | 7.04 |
+---------+--------------+-----------+---------------+

Note: You can see the more details of resource usage below for individual instances under tenant:
See the second indtsnce usages there it's showing local_gb as 1 that means flavor.root_gb is used
even if the instance is booted from volume.

$ curl -g -i -X GET "http://10.232.48.200/compute/v2.1/os-simple-tenant-usage/f77f8f0f6f1b46fe8a43b27a5372f427?start=2017-09-07T00:00:00&end=2017-09-08T06:58:01.961267" -H "OpenStack-API-Version: compute 2.53" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.53" -H "X-Auth-Token: 51a6cb7d32b14bb7b22827f517e3d544"

HTTP/1.1 200 OK
Date: Thu, 07 Sep 2017 07:00:29 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 993
Content-Type: application/json
OpenStack-API-Version: compute 2.53
X-OpenStack-Nova-API-Version: 2.53
Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version
x-openstack-request-id: req-b5f88549-1908-47ad-ba02-f3b174e6ca22
x-compute-request-id: req-b5f88549-1908-47ad-ba02-f3b174e6ca22
Connection: close

{
"tenant_usage": {"total_memory_mb_usage": 3591.7622866488887, "total_vcpus_usage": 7.015160716111111, "start": "2017-09-07T00:00:00.000000", "tenant_id": "f77f8f0f6f1b46fe8a43b27a5372f427", "stop": "2017-09-07T07:00:29.578578", "server_usages": [

{"instance_id": "0280c4ff-6401-477d-93d6-c22077c9f8f1", "uptime": 64741, "started_at": "2017-09-06T12:56:09.000000", "ended_at": "2017-09-07T06:55:10.000000", "memory_mb": 512, "tenant_id": "f77f8f0f6f1b46fe8a43b27a5372f427", "state": "terminated", "hours": 6.919444444444444, "vcpus": 1, "flavor": "m1.tiny", "local_gb": 1, "name": "test"},

{"instance_id": "c46243b7-9034-4847-a9c5-480f5e60f59f", "uptime": 344, "started_at": "2017-09-07T06:54:45.000000", "ended_at": null, "memory_mb": 512, "tenant_id": "f77f8f0f6f1b46fe8a43b27a5372f427", "state": "active", "hours": 0.09571627166666666, "vcpus": 1, "flavor": "m1.tiny", "local_gb": 1, "name": "boot-from-volume"}], "total_hours": 7.015160716111111, "total_local_gb_usage": 7.015160716111111}
}

Tags: api volumes bfv
Changed in nova:
assignee: nobody → Bhagyashri Shewale (bhagyashri-shewale)
Revision history for this message
Takashi Natsume (natsume-takashi) wrote :

Set the status 'In-progress' because this report has an assignee.

Changed in nova:
status: New → In Progress
Revision history for this message
Bhagyashri Shewale (bhagyashri-shewale) wrote :
Download full text (6.5 KiB)

Hi all,

To fix this issues they are three solutions which are mentioned below:

Solution 1:

Check the instance is booted from volume or not by calling the dp api and if it's booted from volume then set the flavor.root_gb as 0.

Drawback:

This causes the perfomance issue while getting usages.

Checked the time required to get 1000 instance usages using simple tenant usage api on solution 1 and on current master but seems that it's taking double time to get usages on Solution 1.

You can see the time difference below:

Time taken to take instance usage on master:

1. $ time nova usage --tenant b844ea0d94cb41d289f70e02bf555d15

+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 1000 | 662.61 | 1.29 | 1.29 |
+---------+--------------+-----------+---------------+

real 0m3.325s
user 0m1.660s
sys 0m0.128s

2. $ time nova usage --tenant b844ea0d94cb41d289f70e02bf555d15

+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 1000 | 662.61 | 1.29 | 1.29 |
+---------+--------------+-----------+---------------+

real 0m3.298s
user 0m1.676s
sys 0m0.120s

3. $ time nova usage --tenant b844ea0d94cb41d289f70e02bf555d15

+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 1000 | 662.61 | 1.29 | 1.29 |
+---------+--------------+-----------+---------------+

real 0m3.372s
user 0m1.672s
sys 0m0.108s

Average time: 3 sec

Time taken to take instances usage on Solution 1:

1. $ time nova usage --tenant b844ea0d94cb41d289f70e02bf555d15

+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 1000 | 662.61 | 1.29 | 0.00 |
+---------+--------------+-----------+---------------+
real 0m7.762s
user 0m1.680s
sys 0m0.128s

2. $ time nova usage --tenant b844ea0d94cb41d289f70e02bf555d15

+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 1000 | 662.61 | 1.29 | 0.00 |
+---------+--------------+-----------+---------------+

real 0m6.922s
user 0m1.644s
sys 0m0.160s

3. $ time nova usage --tenant b844ea0d94cb41d289f70e02bf555d15

+---------+--------------+-----------+---------------+
| Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |
+---------+--------------+-----------+---------------+
| 1000 | 662.61 | 1.29 | 0.00 |
+---------+--------------+-----------+---------------+

real 0m6.839s
user 0m1.672s
sys 0m0.092s

Average time: 7 sec

Solution 2:

Tried to get the usages information from the placement api: There are two placement api we can get the usages info which are mentioned below:

1. get allocations for instance using the placement api GET /allocations/{instance_uuid...

Read more...

Revision history for this message
Matt Riedemann (mriedem) wrote :

See related bug 1469179 which was fixed in Rocky. As of Rocky, we don't schedule based on DISK_GB or report DISK_GB usage for volume-backed servers, so we likely shouldn't report local_gb usage in the simple tenant usage API anymore either.

Changed in nova:
importance: Undecided → Medium
tags: added: api bfv volumes
Revision history for this message
Matt Riedemann (mriedem) wrote :

One issue with fixing this in the simple tenant usage API is we pull potentially a lot of instances out of the database and it would be inefficient to call nova.compute.utils.is_volume_backed on each one. We could stash an is_bfv flag in the instance_system_metadata table but that's also not very efficient, and not something we join on today in that API (since joining on that table is also inefficient).

One thought is if we can add a joinedload column in the instances model so that we can query the DB API asking for this is_bfv joinedload column to be populated which would essentially do, via joins in the DB query, what nova.compute.utils.is_volume_backed already checks, which is:

1. Does the instance have related block_device_mappings records?
2. Does one of those records have boot_index=0?
3. Does that record have destination_type='volume'?

If so, it's a volume-backed server. Then we could just joinedload that column in the DB query, pull it off the instance object in the API and get our flag.

Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/612626

Changed in nova:
assignee: Bhagyashri Shewale (bhagyashri-shewale) → Pooja Jadhav (poojajadhav)
Changed in nova:
assignee: Pooja Jadhav (poojajadhav) → Shilpa Devharakar (shilpasd)
Changed in nova:
assignee: Shilpa Devharakar (shilpasd) → Bhagyashri Shewale (bhagyashri-shewale)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/694462

Changed in nova:
assignee: Bhagyashri Shewale (bhagyashri-shewale) → Shilpa Devharakar (shilpasd)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/694463

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.