[Glance] Glance user storage quota bypass #1

Aleksey Galkin, could you please verify the fix on MOS 6.1 release ISO?

Ok, this bug was based on https://bugs.launchpad.net/glance/+bug/1398830

Tested on MOS 6.1 with 2 controllers and 2 computes.

{"build_id": "2015-06-19_13-02-31", "build_number": "525", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-19_13-02-31", "build_number": "525", "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "dbd54158812033dd8cfd7e60c3f6650f18013a37", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "a3998372183468f56019c8ce21aa8bb81fee0c2f"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "dbd54158812033dd8cfd7e60c3f6650f18013a37", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "a3998372183468f56019c8ce21aa8bb81fee0c2f"}

Steps to reproduce:

1) Set 'file' storage on glance-api.conf.
2) Set 'user_storage_quota' to 604979776 on glance-api.conf (a little more than the size of the image)
3) Restart glance-api service.
4) Run attached script 'test_images.py' on screen.
5) Wait some time (about 10-20 min) and use 'df -h' command.
6) We can see something about this: /dev/mapper/image-glance 246G 77G 169G 32% /var/lib/glance .
7) Glance used 77G, but we set user_storage_quota on ~6G.

Closing this bug as Invalid we consider this is another way of bypassing the quota and the original issue is fixed. We filed another bug to track the new issue - https://bugs.launchpad.net/mos/+bug/1497984 (it's in private security status now)

Sorry - changind the status to Fix Committed

Vitaly - where is the fix that was committed for this bug?

The fix was consumed from stable/kilo

Thanks, Mike!
Tested on iso #301 (RC4).
Your fix works correctly, good job!

Fix proposed to branch: openstack-ci/fuel-5.1.1-updates/2014.1.1
Change author: Zhi Yan Liu <email address hidden>
Review: https://review.fuel-infra.org/13121

Reviewed: https://review.fuel-infra.org/13121
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1.1-updates/2014.1.1

Commit: f7bc56ce85aa7d9f2d7332ba4bb631737789c3d9
Author: Zhi Yan Liu <email address hidden>
Date: Tue Nov 10 10:41:33 2015

Cleanup chunks for deleted image that was 'saving'

Currently image data cannot be removed synchronously for an image that
is in saving state. And when, the upload operation for such an image is
completed the operator configured quota can be exceeded.
This patch fixes the issue of left over chunks for an image which was
deleted from saving status. However, by the limitation of the design we
cannot enforce a global quota check for the image in saving status.
This change introduces a inconsonance between http response codes of
v1 and v2 APIs. The status codes which we will now see after the upload
process completes on an image which was deleted mid way are:

v1: 412 Precondition Failed
v2: 410 Gone

SecurityImpact
UpgradeImpact
APIImpact

Closes-Bug: 1383973
Closes-Bug: 1398830
Closes-Bug: 1188532
Closes-Bug: 1414685

Conflicts:
        glance/api/v1/upload_utils.py
        glance/api/v2/image_data.py
        glance/quota/__init__.py
        glance/tests/unit/test_domain_proxy.py
        glance/tests/unit/test_quota.py
        glance/tests/unit/v1/test_api.py

Signed-off-by: Zhi Yan Liu <email address hidden>
(cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60

Verified on 5.1.1

Test cases are automated: https://review.gerrithub.io/#/c/277346/