PermissionError for lock file

MAAS uses twisted and we have ran into an issue where it seems to be that the MAAS's networking monitoring services has a lock file to ensure that only one processes updates the networking information. If the processes gets killed, the lock file stays, pointing to the PID the killed regiond process had.

Now what normally happens is that another process tries to acquire the lock, sees that the lock points to a killed PID , and recreates the lock.

This normally works, but what can happen is that the killed PID gets recycled, so that the lock now points to a PID which the maas user isn't allowed to kill. Now a PermissionError is raised, that the lock file implementation doesn't handle this case, and the networking monitoring service can never start.

Here is the traceback:

2019-10-31 08:12:13 twisted.scripts: [info] twistd 17.9.0 (/usr/bin/python3 3.6.8) starting up.
2019-10-31 08:12:13 twisted.scripts: [info] reactor class: twisted.internet.asyncioreactor.AsyncioSelectorReactor.
2019-10-31 08:12:14 -: [critical] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/task.py", line 194, in start
            self()
          File "/usr/lib/python3/dist-packages/twisted/internet/task.py", line 239, in __call__
            d = defer.maybeDeferred(self.f, *self.a, **self.kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
            result = f(*args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
            return _inlineCallbacks(None, gen, Deferred())
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
            result = g.send(result)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1001, in updateInterfaces
            responsible = self._assumeSoleResponsibility()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1077, in _assumeSoleResponsibility
            self._lock.acquire()
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/fs.py", line 467, in acquire
            if not self._fslock.lock():
          File "/usr/lib/python3/dist-packages/twisted/python/lockfile.py", line 185, in lock
            kill(int(pid), 0)
        builtins.PermissionError: [Errno 1] Operation not permitted