MAAS_PATH env variable missing from snap env, causes maas not to be able to connect to virsh

Bug #2080646 reported by Nishant Dash
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Triaged
High
Unassigned
3.5
Triaged
High
Unassigned
snapd
New
Undecided
Unassigned

Bug Description

I get this traceback

```
# maas.power on virsh --power-address qemu+ssh://maas@<IP>/system --power-id juju-dash3-test2
main function encountered error
Traceback (most recent call last):
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
    self._startRunCallbacks(fail)
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback( # type: ignore[misc]
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/snap/maas/36889/lib/python3.10/site-packages/provisioningserver/power_driver_command.py", line 89, in _run
    await driver.on(None, context)
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/snap/maas/36889/lib/python3.10/site-packages/provisioningserver/drivers/power/__init__.py", line 384, in perform_power
    yield power_func(system_id, context)
  File "/snap/maas/36889/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/snap/maas/36889/lib/python3.10/site-packages/provisioningserver/drivers/pod/virsh.py", line 1317, in power_control_virsh
    raise VirshError("Failed to login to virsh console.")
provisioningserver.drivers.pod.virsh.VirshError: Failed to login to virsh console
```

At first I though it might be similar to https://bugs.launchpad.net/maas/+bug/2053033 but something different is happening here.

Adding more debugging statements to the virsh.py file via a custom ro mount, I was able to see the context object and see the command it was running
```
args: ['/snap/maas/36889/usr/bin/virsh', '--connect', 'qemu+ssh://maas@<IP>/system?command=/usr/lib/maas/unverified-ssh']
buffer (last 100 chars): b''
before (last 100 chars): b't execute binary /usr/lib/maas/unverified-ssh: No such file or directory: Connection reset by peer\r\n'
```

It can not find `/usr/lib/maas/unverified-ssh` which exists as `/snap/maas/current/usr/lib/maas/unverified-ssh`

There is a get path function which tried to read the env var `MAAS_PATH` to use as the base of the path

setting `MAAS_PATH=/snap/maas/current` works for the immediate commands in the shell. However the only way to get your maas setup to use it would be to hardcode the path in virsh.py and mount that file for your maas snap

Revision history for this message
Nishant Dash (dash3) wrote :

I should add this is maas `3.5.1-16317-g.409891638` revision `36889` on jammy LTS

Changed in maas:
status: New → Triaged
importance: Undecided → High
milestone: none → 3.6.0
Revision history for this message
Andrew Lamzed-Short (andyls) wrote :

This bug initially looked to be a case of adding and populating the correct environment variables to the snapcraft.yaml and calling it a day.

However, during testing, I was encountering this error message when running the `maas.power` command from the reproduction steps:

`libvirt: error : cannot execute binary /usr/bin/pkttyagent: Permission denied`

Connecting to the host manually outside of the snap using `virsh -c qemu+ssh://[...]` was working fine. It appeared that this was only happening within the snap.

Confirmation of this came upon examining /var/log/syslog and seeing that AppArmor was denying virsh exec permissions on pkttyagent for snap.maas.power.

`Nov 11 17:35:02 work-laptop kernel: audit: type=1400 audit(1731346502.225:4828): apparmor="DENIED" operation="exec" class="file" profile="snap.maas.power" name="/usr/bin/pkttyagent" pid=201812 comm="virsh" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0`

virsh (a part of libvirt) relies on PolicyKit (a.k.a. polkit) for authentication purposes. pkttyagent is a binary polkit provides that provides a textual authentication agent for inputting passwords etc. from the terminal. Access to this from a snap perspective is provided by snapd's plug interfaces, specifically the polkit plug. As this issue only occurs in-snap, this looks like it might either be a bug in the polkit plug, or at the very least the plug will require some extension to allow snap developers to modify the functionality or configuration of it as required. This discussion sheds a bit more light on this matter: https://forum.snapcraft.io/t/polkit-interface-and-pkttyagent-in-snap/41616

I suspect a number of other bugs, such as https://bugs.launchpad.net/maas/+bug/2053033, will also likely run into this same situation until this is addressed. This will require a separate snapd bug report which will thus block this issue (notwithstanding any intermediate workarounds).

Changed in maas:
milestone: 3.6.0 → 3.6.x
Revision history for this message
Jacopo Rota (r00ta) wrote :

Adding snapd team for visibility on this bug

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.