Unable to sideload large snap because of not enough space on /tmp

Just letting snapd folks know this is a project we are working with.

Ilya, can you please provide the output of 'snap list' command? thanks. (Feel free to obfuscate any sensitive snap names)

The output of df -h would also be good to see

This error message actually seems to be coming from the go standard library in ReadForm(...), which is before in snapd have even seen the data.

$ snap list
Name Version Rev Tracking Publisher Notes
avnet-avt-iiotg20-kernel 5.4.0-1022.25.4 58 latest/stable canonical✓ kernel
core 16-2.50 11085 latest/stable canonical✓ core
core18 20210507 2067 latest/stable canonical✓ base
core20 20210429 1029 latest/stable canonical✓ base
kdrp-custom-1 1.0.2 3 latest/stable
kdrp-custom-2 20-6 25 latest/stable
kdrp-custom-3 1.1.3-20211011005841.ubuntu20 226 latest/stable
kdrp-custom-4 1.1.3-20211011005841.ubuntu20 219 latest/stable
kdrp-custom-5 develop 95 latest/edge/…
kdrp-custom-6 1.1 3 latest/stable
modem-manager 1.12.8-1 456 20/stable canonical✓ -
mosquitto 2.0.11 631 latest/stable mosquitto✓ -
network-manager 1.22.10-6 662 20/stable canonical✓ -
snapd 2.50 11844 latest/stable canonical✓ snapd
ufw 0.36.1 481 latest/stable canonical✓ -

$ df -h | grep -v kdrp
Filesystem Size Used Avail Use% Mounted on
devtmpfs 354M 0 354M 0% /dev
tmpfs 428M 16K 428M 1% /dev/shm
tmpfs 86M 6.6M 80M 8% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 428M 0 428M 0% /sys/fs/cgroup
/dev/dm-0 13G 4.1G 7.4G 36% /writable
/dev/loop0 52M 52M 0 100% /
/dev/loop1 259M 259M 0 100% /run/mnt/kernel
/dev/dm-1 7.8M 191K 6.9M 3% /var/lib/snapd/save
/dev/mmcblk0p2 739M 56M 684M 8% /run/mnt/ubuntu-boot
/dev/mmcblk0p1 1.6G 1.1G 526M 67% /var/lib/snapd/seed
tmpfs 428M 0 428M 0% /media
tmpfs 428M 0 428M 0% /mnt
tmpfs 428M 4.0K 428M 1% /tmp
tmpfs 428M 0 428M 0% /var/lib/sudo
/dev/loop2 248M 248M 0 100% /snap/avnet-avt-iiotg20-kernel/53
/dev/loop3 85M 85M 0 100% /snap/core/11085
/dev/loop4 46M 46M 0 100% /snap/core18/2067
/dev/loop19 5.0M 5.0M 0 100% /snap/network-manager/662
/dev/loop20 1.2M 1.2M 0 100% /snap/mosquitto/631
/dev/loop21 27M 27M 0 100% /snap/snapd/11844
/dev/loop22 768K 768K 0 100% /snap/ufw/481
/dev/loop23 1.7M 1.7M 0 100% /snap/modem-manager/456
tmpfs 86M 0 86M 0% /run/user/1000

Looking at Go stdlib, mime/multipart.Reader.readFrom() creates a temporary file in the default tempdir (/tmp) when the size of a part is larger than 1MB (a limit set by snapd). This is a bit unfortunate, as snapd then creates another temporary file to which we copy he contents under /var/lib/snapd/snap, so we end up with 2 copies of the same thing (at least temporarily).

I think a workaround could be to set TMPDIR to some location with enough free space to store the snap. Setting it in /etc/enviromnent is not the best choice as it makes it system wide, but an override file for snapd.service (eg. /etc/systemd/system/snapd.service.d/tmpdir.conf) with Environment=TMPDIR=<some-location> should be sufficient.

On the snapd side, the code probably needs to handle such requests looping over NextPart() rather than using stdlib provided wrapper.

Why is this "incomplete"? The project is seriously affected by this (they have temporarily decreased the size of the snap that won't install, but that's not a viable/stable workaround). Does the proposed workaround in comment #7 apply to ubuntu core 20 images -- I don't see an interface that allows writing there?

Hi Kyle, this was marked as Incomplete because that is part of our bug handling process. When we need more information or ask more details on a bug, we will set it to Incomplete and the expectation is that when the reporter provides this information, the bug is set back to New so it is picked up by us when we do triage.

The proposed workaround would work on UC20 but would need to be done from outside confinement indeed, there is no interface that would allow doing that from a strict snap. This was mentioned just as a workaround not as a real solution.

To be clear, the only real long-term way to fix this is to change snapd's implementation as Maciej suggested.

We need a workaround solution that can work from inside confinement in the context of a snap distributed on a closed UC 20 platform from a private brand store. This issue has the potential to block a major product release and is a very high priority for us to address.

Can we use the system-files interface to access the /etc/systemd/system/snapd.service.d/tmpdir.conf and programmatically modify it as proposed?

https://github.com/snapcore/snapd/pull/11053

Hi. The project asked a question a week ago in comment #7. Is that a feasible workaround since the fix is not landed yet in snapd? Would the snapd team allow this via system-files, and indeed would it work? Thanks, Kyle

I am told that the system-files workaround would NOT be granted, so we need to get the project timeline clearly stated so snapd team can understand.

This will be in snapd 2.53.3, which we will be preparing soon in the beta channel, but the fix is already available on edge, can folks test the edge channel of snapd to make sure this is fixed for you?

Thanks

We tested and the fix seems to work. Thank you. We are looking forward to having the fix on latest/stable.

Thanks for validating it, we hope to have 2.53.4 on beta by the end of the week (we had to skip 2.53.3 for unrelated reasons)