Different reclaim ages for accounts and containers can result in un-reclaimable containers

I don't understand. Why aren't the container's reported_ timestamps getting updated during the reclaim period? The account server should respond 2XX to PUT container requests with x-account-override-deleted. Is the container-updater not able to cycle within the account's reclaim age?

If we're trying to handle the container-updater hitting an account that has already been reclaimed then the account server needs to catch the Exception coming out of account_broker.put_container and handle it. I think that it should be the job of the container-updater to recognize when is the right time to "give up" on updating the account's. For example; perhaps when the majority of account servers respond 404 (or whatever response is returned from the exception handler for put_container) and broker.is_deleted and time() - delete_timestamp > reclaim_age.

> Why aren't the container's reported_ timestamps getting updated during the reclaim period?

They are reported. Lets say that reclaim_age for accounts is 6 days and containers are usual 7 days. When the account is deleted, the container-updater has 6 days during which it can push the timestamps to the account. It almost certainly manages to do this.

On day 6, the account database is reclaimed.

On day 7, three container-replicators are due to reclaim/delete the database files. My supposition is that one (node A) is slightly ahead of the other two and deletes the database. Since the other servers (B and C) are slightly behind, they have not reached their reclaim_age so one of the container-replicators rsyncs the database back to A. As part of this process, the reported_*_timestamps are set to zero (see container.backend._newid). Minutes later, B and C reclaim their database files.

Normally, this race would resolve itself, because container-updater on A would push the timestamps to the account and set reported_*_timstamps. However, in this scenario the account was deleted the day before, so container-updater never succeeds.

I don't have logs to prove all of this -- I'm using a mental exercise to work out how we got into this state.

> ...then the account server needs to catch the Exception coming out of account_broker.put_container and handle it...

I won't disagree with that. By not handling the exception, the account-server ends up reporting e500 instead of e404. However, I regarded that as a second order problem. I would need to spend more time to decide if 404 meant something to other components before proposing to fix that. As it happens, the e500 was useful to find this problem because we noticed that some nodes has higher e500 counts than other nodes.

> ....it should be the job of the container-updater to recognize when is the right time to "give up" on updating the account's

This would stop the e500 noise in the system. However, we end up with an orphaned db file that's never deleted. By deleting the db in container-replicator, we stop container-updater stumbling across such files.

Note: I have a proposed fix here: https://review.openstack.org/#/c/84696/ I don't know why gerrit didn't update this bug (mabe because it's WIP).

I don't think "giving up" on updating is a good idea, because it may cause danger that
account will never be reclaimed when all container-server fail to update and give up.

> As part of this process, the reported_*_timestamps are set to zero (see container.backend._newid).
I think this is the root of this problem.
When database is completely rsynced from container-server B to container-server A,
we don't have to reset reported_*_timestamp at container-server A
because container-server B have updated the account.
# We have to reset it only after merging databases.

I think removing that reset processing is a solution for this problem.
It stops container-server A from trying this unnessesory updating after replication,
and then replicated container will be reclaimed normally.

>it may cause danger that account will never be reclaimed when all container-server fail to update and give up.

I'm proposing to delete the container databases, not give up on contianer-updater. However, I take your point: if we delete the container database, the account might not get reclaimed. However, to get into this state, all three copies of the account must be down for a month (or conversely, the container db must be off line for a month). The upshot either way is that *some* database file becomes un-reclaimable.

> As part of this process, the reported_*_timestamps are set to zero (see container.backend._newid).
>> I think this is the root of this problem.

I'm reluctant to change that -- there must be good reason for the __newid() function. This is also used when you do a merge of records (vs. complete file copy)

In addition, this would only prevent future un-reclaimable databases -- not clean out my existing database files.

Can I suggest that you make comments against the proposed change at: https://review.openstack.org/#/c/84696
The Swift core developers tend to interact more in gerret than in launchpad

Would something like the container-replicator only replicator if the account exists (so an account check) solve this in this very specific use case. Existing problem containers or a storage node coming back online with older reported_*_timestamps would still have the issue however. Quarantining makes sense.

Would it be beneficial to set a more conservative values for reclaim_age?

If account's reclaim_age > container's reclaim_age> objects' reclaim_age, that should get rid of the race condition, shouldn't it?

Also, what would be the best way of getting rid of the unclaimed containers? Just delete them or is there a better way to let the cluster do it?

@Juan

Yes, at a *minimum* we should document best practices on the reclaim ages of objects, containers & accounts!

Yes, I would expect if traversing the container tree on all nodes is a viable option in your cluster removing the database (or moving to an out of the datadir directory) would allow everything to process cleanly.

... although fixing the bug would be even better <grrr>

There's some concern that entirety of the problem and the ramifications aren't fully understood - but if you recently encountered this situation in a production environment you might be able to provide additional forensics? Like what settings you were running for the reclaim ages - how many containers you found in this state - if all replicas of the containers reflected the same last update values - anything else that seems relevant...

Sorry for the delayed response, apparently I didn't subscribe to this bug report.

@clayg (clay-gerrard): we had the defaults for those values and we have experienced the problem of un-reclaimable containers in one of our production clusters, that we've been running since 2011. I guess that being live longer increases the chances of the race condition happening, as more accounts have been deleted than in more recent clusters.

After a quick look I've counted 6 containers on that cluster, but I need to do a proper analysis of the logs to be 100% sure.

I agree that a proper fix would be better. We changed reclaim age as I explained in my previous comment, but it is hard to tell if it has solved the problem because we don't have that many accounts deleted and being a race condition it may or may not happen (even if the new configuration didn't solve the problem).

@juan np, I'm sure that will be helpful info when someone dives in. Thanks.

If you're still digging in feel free to jump in #openstack-swift on Freenode and ask if any swift junkies for help - we're pretty easy to nerd snipe.