Comment 0 for bug 1698304

Revision history for this message
Mark Kirkwood (mark-kirkwood) wrote : swiift-container-info explodes when investigating quarantined containers

This analysis was performed on Swift 2.7.0.

Suppose a container is quarantined. One of the fist things you want to do is check where it came from initially. So reach for swift-container-info. It turns out this is a bad idea:

$ cd /srv/node/vdc/quarantined/containers/3a7b4bae41a17d0f54b247c727b4f0cd/
$ ls -l
total 20
-rw------- 1 swift swift 18432 Aug 15 2016 3a7b4bae41a17d0f54b247c727b4f0cd.db
$ sudo swift-container-info ./3a7b4bae41a17d0f54b247c727b4f0cd.db
Traceback (most recent call last):
  File "/usr/bin/swift-container-info", line 32, in <module>
    print_info('container', *args, **vars(options))
  File "/usr/lib/python2.7/dist-packages/swift/cli/info.py", line 327, in print_info
    info = broker.get_info()
  File "/usr/lib/python2.7/dist-packages/swift/container/backend.py", line 501, in get_info
    with self.get() as conn:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 366, in get
    self.possibly_quarantine(*sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 347, in possibly_quarantine
    renamer(self.db_dir, quar_path, fsync=False)
  File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer
    os.rename(old, new)
OSError: [Errno 16] Device or resource busy

ok that was bad - not only do we not get any useful info but it has tried to re-quarantine the container (and in the process mv the directory I am in...yikes, and hence the error):

$ ls -l
total 20
-rw------- 1 swift swift 18432 Aug 15 2016 3a7b4bae41a17d0f54b247c727b4f0cd.db
drwxr-xr-x 3 root root 24 Jun 16 06:55 quarantined

This comes about because swift-container-info:
- calls print_info which creates a ContainerBroker and...
- calls ContainerBroker.get_info using a self.get which resolves to DatabaseBroker.get
- DatabaseBroker.get calls self.possibly_quarantine

So the innocuous 'get_info' has dramatic side effects (possibly_quarantine)

We probably need to amend get_info e.g get_info(no_quarantine) to be side effect free for this type of use!