snapcraft validate (and possibly other operations requiring gpg passphrases) fail on remote/headless systems

A customer reported (and I was able to reproduce) getting an error when using snapcraft validate.

The key to reproducing (for me at least - still awaiting confirmation from customer) was doing this on a remote headless system. Sounds like gpg is trying to invoke something that isn't present on the system to get the passphrase and then fails. But it seems to happen only when gpg is invoked by snapcraft - so perhaps gpg is detecting something about its running environment or stdin/out redirection when running under snapcraft, and trying to ask for the password differently, and one thing to do would be forcing gpg to ask for the password a different way or something.

To reproduce:

1- ssh into a system that has snapcraft. I used a VM installed from an ubuntu server cloud image.

snapcraft validate allyoursnaparebelongtous1 allyoursnaparebelongtous2=2 --key-name an-example-three
Getting details for allyoursnaparebelongtous2
Signing validations assertion for allyoursnaparebelongtous2=2
Error signing validations assertion for allyoursnaparebelongtous2=2: error: cannot sign assertion: cannot sign using GPG: /usr/bin/gpg --personal-digest-preferences SHA512 --default-key 0x4A7677B256FB90F61B91C061C3E6F877EC048DD3 --detach-sign failed: exit status 2 ("gpg: signing failed: No such file or directory\ngpg: signing failed: No such file or directory\n")

The workaround is to first run the gpg command outside of snapcraft, so the key is unlocked:

echo "lalal" > a-file
/usr/bin/gpg --homedir=.snap/gnupg --personal-digest-preferences SHA512 --default-key 0xE185A8404D4CCDA151FA4AC51F5108DA9CD52BC6 --detach-sign a-file

then the snapcraft operation succeeds:

snapcraft validate allyoursnaparebelongtous1 allyoursnaparebelongtous2=2 --key-name an-example-two
Getting details for allyoursnaparebelongtous2
Signing validations assertion for allyoursnaparebelongtous2=2

snapcraft version
snapcraft, version 4.5.4

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

 snap version
snap 2.49
snapd 2.49
series 16
ubuntu 18.04
kernel 4.15.0-136-generic

This is likely related to:

https://forum.snapcraft.io/t/snap-sign-unable-to-invoke-gpg-agent-pinentry/10938

and:

https://bugs.launchpad.net/snapcraft/+bug/1866257

from the latter, "there is a known issue with gpg-agent not freeing the lock"