ppa will fail to install under proxy environment

At this revision, we lost an ability of installing from a mirrored repository:
https://git.launchpad.net/landscape-charm/commit/config.yaml?h=main&id=01f72f99b84fcf8e7c8eef81883abb12209e3461

Installation from a mirror server can happen where a child Landscape doesn't have internet access but to fetch packages from a parent Landscape in DMZ in some enterprise requirements.

Subscribing ~field-high. Looks like we lost some of the abilities for customer deployment in the charm rewrite.

I've tested with these additional juju model-config settings. Please confirm if you have these ones set as well, meaning apt-http-proxy, apt-https-proxy. My local proxy is at 10.76.244.153:8080 and I can confirm traffic crossing the proxy:

apt-ftp-proxy model http://10.76.244.153:8080/
apt-http-proxy model http://10.76.244.153:8080/
apt-https-proxy model http://10.76.244.153:8080/
apt-mirror default ""
apt-no-proxy default ""
ftp-proxy model http://10.76.244.153:8080/
http-proxy model http://10.76.244.153:8080/
https-proxy model http://10.76.244.153:8080/
no-proxy default 127.0.0.1,localhost,::1

Sample of traffic logs from proxy during juju deploy:

10.76.244.203 - - [06/Jul/2023:21:08:37 +0000] "CONNECT api.launchpad.net:443 HTTP/1.1" 200 215160 "-" "-"
10.76.244.203 - - [06/Jul/2023:21:08:45 +0000] "GET http://archive.ubuntu.com/ubuntu/pool/main/l/language-pack-en-base/language-pack-en-base_22.04%2b20230209_all.deb HTTP/1.1" 200 438391 "-" "Debian APT-HTTP/1.3 (2.4.9) non-interactive"

Of course, this is just testing with a proxy, not in a DMZ.

Installation succeeds and the landscape-server application reaches the status "Waiting on relations"

The thing is setting http-proxy globally in one model has side effects. That's why juju-http-proxy was born to give a more granular control over what traffic should be proxied or not.

The legacy charm had http-proxy and https-proxy in the charm config itself so it was okay. And the new charm has http_proxy and https_proxy but I don't see an implementation in the charm to leverage those values at all.

So the issue here is there is no way to use proxy if we use juju-http-proxy instead of the legacy model global http-proxy which is full of side-effect.

I cannot find the mention of "legacy" or "deprecated" in the official doc any longer somehow. But it's clearly mentioned as LegacyProxySettings.

https://github.com/juju/juju/blob/0cbfebb898b895ca95016df3ee09e6c105ed1b63/environs/config/config.go#L989-L1028

Here's the settings I used. This works with the current edge version of the charm:

apt-ftp-proxy model http://10.76.244.153:8080/
apt-http-proxy model http://10.76.244.153:8080/
apt-https-proxy model http://10.76.244.153:8080/
apt-mirror default ""
apt-no-proxy default ""
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
no-proxy default 127.0.0.1,localhost,::1
juju-ftp-proxy model http://10.76.244.153:8080/
juju-http-proxy model http://10.76.244.153:8080/
juju-https-proxy model http://10.76.244.153:8080/
juju-no-proxy default 127.0.0.1,localhost,::1

This is using both the apt-*-proxy settings and the juju-*-proxy settings and works fine for me. Please confirm (as I asked previously) that the apt-*-proxy settings are set, because these are used for add-apt-repository and other apt-related networking.

I believe that all of the necessary proxy configs are set in model-config.
Here are the proxy related configs I have set.

apt-http-proxy: http://192.168.1.28:8000
apt-https-proxy: http://192.168.1.28:8000
snap-http-proxy: http://192.168.1.28:8000
snap-https-proxy: http://192.168.1.28:8000
juju-http-proxy: http://192.168.1.28:8000
juju-https-proxy: http://192.168.1.28:8000
no-proxy: 127.0.0.1,localhost,192.168.1.0/24

We are deploying a full LMA stack in this model, which includes landscape, elasticsearch, graylog, grafana, prometheus, etc.
I can confirm other applications are succeeding with adding ppa and installing apt packages, so I believe the proxy configuration are okay.

@mitchburton, thanks for looking into. We will mark this as Incomplete for the time being as we need a bit more testing on our side.

Hmm, it was fairly straightforward to reproduce on a clean environment.

- MAAS provider (192.168.151.0/24 is the network managed by MAAS and 192.168.151.1 is MAAS)
- Juju 2.9.43-ubuntu-amd64
- landscape-dense-maas bundle from the charmstore

cat <<EOF | tee cloudinit-userdata.yaml
apt-http-proxy: http://192.168.151.1:8000/
apt-https-proxy: http://192.168.151.1:8000/
juju-http-proxy: http://192.168.151.1:8000/
juju-https-proxy: http://192.168.151.1:8000/
juju-no-proxy: 127.0.0.1,localhost,::1,192.168.151.0/24
snap-http-proxy: http://192.168.151.1:8000/
snap-https-proxy: http://192.168.151.1:8000/

cloudinit-userdata: |
  preruncmd:
  - ufw default allow # don't block anything by default
  - ufw reject out http # block outgoing 80/tcp connection without proxy
  - ufw reject out https # block outgoing 443/tcp connection without proxy
  - ufw enable
EOF

juju add-model landscape-test --config cloudinit-userdata.yaml

juju deploy landscape-dense-maas && time juju-wait -w

-> ERROR:root:landscape-server/0 failed: workload status is error

[juju debug-log -i landscape-server/0 --replay]
unit-landscape-server-0: 11:03:00 DEBUG unit.landscape-server/0.install Cannot add PPA: 'ppa:~landscape/ubuntu/self-hosted-beta'.
unit-landscape-server-0: 11:03:00 DEBUG unit.landscape-server/0.install ERROR: '~landscape' user or team does not exist.

$ juju ssh landscape-server/0 -- sudo ufw status verbose
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), deny (routed)
New profiles: skip

To Action From
-- ------ ----
80/tcp REJECT OUT Anywhere
443/tcp REJECT OUT Anywhere
80/tcp (v6) REJECT OUT Anywhere (v6)
443/tcp (v6) REJECT OUT Anywhere (v6)

I also tried latest/edge landscape charm and it failed as well.

unit-landscape-server-0: 11:44:21 DEBUG unit.landscape-server/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-landscape-server-0/charm/.unit-state.db.
unit-landscape-server-0: 11:44:21 DEBUG unit.landscape-server/0.juju-log Emitting Juju event install.
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install Traceback (most recent call last):
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/bin/add-apt-repository", line 364, in <module>
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install sys.exit(0 if addaptrepo.main() else 1)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/bin/add-apt-repository", line 347, in main
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install shortcut = handler(source, **shortcut_params)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/shortcuts.py", line 40, in shortcut_handler
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install return handler(shortcut, **kwargs)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 82, in __init__
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install if self.lpppa.publish_debug_symbols:
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 120, in lpppa
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self._lpppa = self.lpteam.getPPAByName(name=self.ppaname)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 107, in lpteam
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self._lpteam = self.lp.people(self.teamname)
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 98, in lp
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self._lp = login_func("%s.%s" % (self.__module__, self.__class__.__name__),
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 494, in login_anonymously
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install return cls(
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 230, in __init__
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install super(Launchpad, self).__init__(
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install File "/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 472, in __init__
unit-landscape-server-0: 11:44:22 WARNING unit.landscape-server/0.install self....

I also hit that bug, but I got an error related to the timeout (similar to the description of this bug), not network unreachable. When i do debug-hook and run the hook manually, it worked correctly.

We've hit this problem as well on the current stable branch, and will be trying to work around it by deploying landscape and dependencies to a separate "landscape" model where we can set the legacy-style proxy settings with limited impact to other apps.

Also, it's worth noting that this issue is kind of "hidden". What I mean is: the charm is effectively broken once the add-apt-repository call fails, yet that just causes a blocked status and it's not until several hooks later that things "really" break. We happened to check "juju show-status-log $unit" and could find the root cause there.

For the record: using the legacy proxy settings seems to allow for a clean deploy. However, if I then try to deploy landscape-client, the proxy settings seem to get in the way. It appears the environment-level proxy settings are then stored in /etc/landscape/client.conf (without any "no-proxy"-type setting), and this ends up breaking things when that proxy shouldn't be used to reach landscape. I also then have to manually edit the /etc/landscape/client.conf files to remove such proxy references in order for registration to succeed.

On the other hand, the juju-*-proxy settings do not result in such config settings in the landscape-client charm.

Long story short: the current implementation is problematic if the customer requires a proxy for external access, yet the proxy shouldn't be used for accessing landscape via landscape-client. If we could avoid needing to use the legacy proxy settings, it would help.

I am hitting this bug as well, jammy 22.04.5
landscape-server charm latest/stable REV 121

juju model-config RE: proxy:

apt-ftp-proxy default ""
apt-http-proxy controller http://172.19.0.10:8000
apt-https-proxy controller http://172.19.0.10:8000
apt-no-proxy default ""
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
juju-ftp-proxy default ""
juju-http-proxy controller http://172.19.0.10:8000
juju-https-proxy controller http://172.19.0.10:8000
juju-no-proxy controller 10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,127.0.0.1,localhost
no-proxy default 127.0.0.1,localhost,::1
snap-http-proxy controller http://172.19.0.10:8000
snap-https-proxy controller http://172.19.0.10:8000

I also have, via cloudinit-userdata, setting the local proxy settings in /etc/enviroment to be:
write_files:
- content: |
    PATH="/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
    LC_ALL="en_US.UTF-8"
    LANG="en_US.UTF-8"
    LANGUAGE="en_US:en"
    EDITOR="vim"
    https_proxy=http://172.19.0.10:8000
    HTTPS_PROXY=http://172.19.0.10:8000
    HTTP_PROXY=http://172.19.0.10:8000
    http_proxy=http://172.19.0.10:8000
    no_proxy="localhost,.0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,.86,.87,.88,.89,.90,.91,.92,.93,.94,.95,.96,.97,.98,.99,.100,.101,.102,.103,.104,.105,.106,.107,.108,.109,.110,.111,.112,.113,.114,.115,.116,.117,.118,.119,.120,.121,.122,.123,.124,.125,.126,.127,.128,.129,.130,.131,.132,.133,.134,.135,.136,.137,.138,.139,.140,.141,.142,.143,.144,.145,.146,.147,.148,.149,.150,.151,.152,.153,.154,.155,.156,.157,.158,.159,.160,.161,.162,.163,.164,.165,.166,.167,.168,.169,.170,.171,.172,.173,.174,.175,.176,.177,.178,.179,.180,.181,.182,.183,.184,.185,.186,.187,.188,.189,.190,.191,.192,.193,.194,.195,.196,.197,.198,.199,.200,.201,.202,.203,.204,.205,.206,.207,.208,.209,.210,.211,.212,.213,.214,.215,.216,.217,.218,.219,.220,.221,.222,.223,.224,.225,.226,.227,.228,.229,.230,.231,.232,.233,.234,.235,.236,.237,.238,.239,.240,.241,.242,.243,.244,.245,.246,.247,.248,.249,.250,.251,.252,.253,.254,.255,.svc,127.0.0.1"
    NO_PROXY="localhost,.0,.1,.2,.3,.4,.5,.6,.7,.8,.9,.10,.11,.12,.13,.14,.15,.16,.17,.18,.19,.20,.21,.22,.23,.24,.25,.26,.27,.28,.29,.30,.31,.32,.33,.34,.35,.36,.37,.38,.39,.40,.41,.42,.43,.44,.45,.46,.47,.48,.49,.50,.51,.52,.53,.54,.55,.56,.57,.58,.59,.60,.61,.62,.63,.64,.65,.66,.67,.68,.69,.70,.71,.72,.73,.74,.75,.76,.77,.78,.79,.80,.81,.82,.83,.84,.85,...