manila_tempest_tests.tests.scenario.test_share_extend fails with CephFS

Bug #2075981 reported by Goutham Pacha Ravi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
In Progress
Low
Goutham Pacha Ravi

Bug Description

Description
===========
The scenario test in the manila_tempest_tests.tests.scenario.test_share_extend module fails in two CephFS CI jobs. The failure was noted since using Ceph 18.2.4 in the CI system.

The test fails at a stage after creating a share and attempting to write past its size:

https://opendev.org/openstack/manila-tempest-plugin/src/commit/a0420244e3d14f4441adab462fe9d2a0f1316b58/manila_tempest_tests/tests/scenario/test_share_extend.py#L88-L98

The size in Ceph is a quota; and it is enforced by the ceph client rather than the server and the ceph documentation states:

"""
Once processes that write data to the file system reach the configured limit, a short period of time elapses between when the amount of data reaches the quota limit, and when the processes stop writing data. The time period generally measures in the tenths of seconds. However, processes continue to write data during that time. The amount of additional data that the processes write depends on the amount of time elapsed before they stop.
"""

So, there's possibly a sync delay where the client is able to write more data than is permitted.

The failure isn't occurring on the cephadm based NFS job (yet)

Ceph Jobs:
"Standalone" NFS: https://zuul.opendev.org/t/openstack/builds?job_name=devstack-plugin-ceph-cephfs-nfs&branch=master&change=915212&skip=0
Cephadm based CephFS/Native: https://zuul.opendev.org/t/openstack/builds?job_name=manila-tempest-plugin-cephfs-native
Cephadm based CephFS/NFS: https://zuul.opendev.org/t/openstack/builds?job_name=manila-tempest-plugin-multinode-cephfs-nfs-cephadm

Steps to reproduce
==================

A chronological list of steps which will help reproduce the issue you hit:
* Create a CephFS share of size 1 GiB (Native or with "Standalone" NFS-Ganesha)
* Fill up the share, write past 1 GiB

Expected result
===============
You're prevented to write past 1GiB

Actual result
=============
No failure occurs

Environment
===========
1. Exact version of OpenStack Manila you are running: master
2. Which storage backend did you use? Ceph, Reef (CephFS-Native)

Tags: cephfs tempest
Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :
Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :
Vida Haririan (vhariria)
tags: added: temp
tags: added: cephfs tempest
removed: temp
Changed in manila:
assignee: nobody → Goutham Pacha Ravi (gouthamr)
milestone: none → dalmatian-rc1
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila-tempest-plugin (master)
Changed in manila:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila-tempest-plugin (master)

Reviewed: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/926004
Committed: https://opendev.org/openstack/manila-tempest-plugin/commit/b758fb49cc8c9d56fa83f910f3462a845eec1b1b
Submitter: "Zuul (22348)"
Branch: master

commit b758fb49cc8c9d56fa83f910f3462a845eec1b1b
Author: Goutham Pacha Ravi <email address hidden>
Date: Fri Aug 2 14:52:13 2024 -0700

    Skip test_share_extend in CephFS jobs

    There's a regression in Ceph tracked by
    https://tracker.ceph.com/issues/67323

    So we'll skip the test for now while the
    issue is resolved in Ceph.

    Change-Id: I030fe57608f4082a84ff60c31132c9bc7cbd1b1e
    Partial-Bug: #2075981
    Signed-off-by: Goutham Pacha Ravi <email address hidden>

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.