failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers

juju 2.8.9 bootstrap failed:

2021-02-24-14:05:15 root ERROR [localhost] Command failed: juju bootstrap --bootstrap-constraints tags=juju --config bootstrap-timeout=1800 --bootstrap-series focal --model-default /home/ubuntu/project/generated/model_defaults.yaml maas_cloud foundations-maas
2021-02-24-14:05:15 root ERROR [localhost] STDOUT follows:
Creating Juju controller "foundations-maas" on maas_cloud/default
Looking for packaged Juju agent version 2.8.9 for amd64
Launching controller instance(s) on maas_cloud/default...
 - hcyhbs (arch=amd64 mem=4G cores=2)
Installing Juju agent on bootstrap instance
Fetching Juju Dashboard 0.3.0
Waiting for address
Attempting to connect to 10.244.40.100:22
Connected to 10.244.40.100
Running machine configuration script...
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'init-local' at Wed, 24 Feb 2021 13:57:30 +0000. Up 10.25 seconds.
2021-02-24 13:57:30,187 - handlers.py[WARNING]: failed posting event: start: init-local/check-cache: attempting to read from cache [trust]
2021-02-24 13:57:30,189 - handlers.py[WARNING]: failed posting event: finish: init-local/check-cache: SUCCESS: no cache found
2021-02-24 13:57:30,507 - handlers.py[WARNING]: failed posting event: finish: init-local: SUCCESS: searching for local datasources
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'init' at Wed, 24 Feb 2021 13:57:32 +0000. Up 12.97 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+----------------------------+---------------+--------+-------------------+
ci-info: | ens4 | True | 10.244.40.100 | 255.255.248.0 | global | 52:54:00:c7:31:b1 |
ci-info: | ens4 | True | fe80::5054:ff:fec7:31b1/64 | . | link | 52:54:00:c7:31:b1 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+----------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.244.40.1 | 0.0.0.0 | ens4 | UG |
ci-info: | 1 | 10.244.40.0 | 0.0.0.0 | 255.255.248.0 | ens4 | U |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | 1 | fe80::/64 | :: | ens4 | U |
ci-info: | 3 | local | :: | ens4 | U |
ci-info: | 4 | ff00::/8 | :: | ens4 | U |
ci-info: +-------+-------------+---------+-----------+-------+
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'modules:config' at Wed, 24 Feb 2021 13:58:05 +0000. Up 46.55 seconds.
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Get:5 http://archive.ubuntu.com/ubuntu focal/universe Translation-en [5124 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/multiverse Translation-en [104 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/main Translation-en [506 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/restricted Translation-en [6212 B]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [155 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [5508 B]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [200 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [23.2 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-security/universe Translation-en [77.3 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-security/multiverse Translation-en [3160 B]
Get:15 http://archive.ubuntu.com/ubuntu focal-security/main Translation-en [111 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-security/restricted Translation-en [19.4 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [1448 B]
Fetched 6336 kB in 5s (1246 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  ifupdown
The following NEW packages will be installed:
  bridge-utils
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 30.5 kB of archives.
After this operation, 112 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 bridge-utils amd64 1.6-2ubuntu1 [30.5 kB]
Fetched 30.5 kB in 0s (72.0 kB/s)
Selecting previously unselected package bridge-utils.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71214 files and directories currently installed.)
Preparing to unpack .../bridge-utils_1.6-2ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.6-2ubuntu1) ...
Setting up bridge-utils (1.6-2ubuntu1) ...
Processing triggers for man-db (2.9.1-1) ...
+ mkdir -p /var/lib/juju
+ cat
+ chmod 0755 /var/lib/juju/MAASmachine.txt
+ set -xe
+ install -D -m 644 /dev/null /var/lib/juju/nonce.txt
+ printf %s\n user-admin:bootstrap
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 running 'modules:final' at Wed, 24 Feb 2021 13:58:06 +0000. Up 48.27 seconds.
Cloud-init v. 20.4.1-0ubuntu1~20.04.1 finished at Wed, 24 Feb 2021 13:58:17 +0000. Datasource DataSourceMAAS [http://10-244-40-0--21.maas-internal:5248/MAAS/metadata/]. Up 59.05 seconds
Logging to /var/log/cloud-init-output.log on the bootstrap machine
Regenerating SSH RSA host key
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:I1robPQRCoSvUdXFP/H8niNm9sRfvRd+hTAFxC0D3zg root@juju-1
The key's randomart image is:
+---[RSA 3072]----+
| ..... o. .+oo |
|... . . ..+oo |
| o. . . +E+. |
|. .. o . o =. |
| o + + S . + . |
|. + + o . .o.o|
| = . .+.=|
| . =.+o=|
| + o.o+|
+----[SHA256]-----+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub
The key fingerprint is:
SHA256:aRkbVEieCCAuz9mgBvWZz3U1zBDEa3Vkc2hpd3H+u8I root@juju-1
The key's randomart image is:
+---[DSA 1024]----+
|. o.. .o===o.+++|
|.o . + +....+o*++|
|o.. + . * .o + .o|
|o+ + o . Bo .|
|..+ . o S. .|
|. . .|
| . . |
| E .|
| .. |
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:ZGeZS1Vkgi1Uiqx4yqkRkn6Whj5Y5LE2IY3Vdz7ckJM root@juju-1
The key's randomart image is:
+---[ECDSA 256]---+
| . +.++o+ |
| . . ..E.o=.o |
| + . +=+B. |
|o.= . +++.. |
|o+.+. o S.. |
|..Bo.+ |
|.=.*+ |
|o.+o |
| .o |
+----[SHA256]-----+
Running apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...
Running apt-get upgrade
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  bind9-dnsutils bind9-host bind9-libs libldap-2.4-2 libldap-common libssl1.1
  open-vm-tools openssl pollinate
9 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 3987 kB of archives.
After this operation, 114 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.2 [1319 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 openssl amd64 1.1.1f-1ubuntu2.2 [620 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 bind9-dnsutils amd64 1:9.16.1-0ubuntu2.6 [134 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 bind9-libs amd64 1:9.16.1-0ubuntu2.6 [1114 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 bind9-host amd64 1:9.16.1-0ubuntu2.6 [43.0 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libldap-common all 2.4.49+dfsg-2ubuntu1.7 [17.1 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libldap-2.4-2 amd64 2.4.49+dfsg-2ubuntu1.7 [155 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 pollinate all 4.33-3ubuntu1.20.04.1 [22.6 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 open-vm-tools amd64 2:11.1.5-1~ubuntu20.04.2 [561 kB]
Preconfiguring packages ...
Fetched 3987 kB in 5s (800 kB/s)
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71242 files and directories currently installed.)
Preparing to unpack .../0-libssl1.1_1.1.1f-1ubuntu2.2_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.2) over (1.1.1f-1ubuntu2.1) ...
Preparing to unpack .../1-openssl_1.1.1f-1ubuntu2.2_amd64.deb ...
Unpacking openssl (1.1.1f-1ubuntu2.2) over (1.1.1f-1ubuntu2.1) ...
Preparing to unpack .../2-bind9-dnsutils_1%3a9.16.1-0ubuntu2.6_amd64.deb ...
Unpacking bind9-dnsutils (1:9.16.1-0ubuntu2.6) over (1:9.16.1-0ubuntu2.4) ...
Preparing to unpack .../3-bind9-libs_1%3a9.16.1-0ubuntu2.6_amd64.deb ...
Unpacking bind9-libs:amd64 (1:9.16.1-0ubuntu2.6) over (1:9.16.1-0ubuntu2.4) ...
Preparing to unpack .../4-bind9-host_1%3a9.16.1-0ubuntu2.6_amd64.deb ...
Unpacking bind9-host (1:9.16.1-0ubuntu2.6) over (1:9.16.1-0ubuntu2.4) ...
Preparing to unpack .../5-libldap-common_2.4.49+dfsg-2ubuntu1.7_all.deb ...
Unpacking libldap-common (2.4.49+dfsg-2ubuntu1.7) over (2.4.49+dfsg-2ubuntu1.6) ...
Preparing to unpack .../6-libldap-2.4-2_2.4.49+dfsg-2ubuntu1.7_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.49+dfsg-2ubuntu1.7) over (2.4.49+dfsg-2ubuntu1.6) ...
Preparing to unpack .../7-pollinate_4.33-3ubuntu1.20.04.1_all.deb ...
Unpacking pollinate (4.33-3ubuntu1.20.04.1) over (4.33-3ubuntu1) ...
Preparing to unpack .../8-open-vm-tools_2%3a11.1.5-1~ubuntu20.04.2_amd64.deb ...
Unpacking open-vm-tools (2:11.1.5-1~ubuntu20.04.2) over (2:11.0.5-4) ...
Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.2) ...
Setting up libldap-common (2.4.49+dfsg-2ubuntu1.7) ...
Setting up pollinate (4.33-3ubuntu1.20.04.1) ...
Setting up open-vm-tools (2:11.1.5-1~ubuntu20.04.2) ...
Installing new version of config file /etc/vmware-tools/tools.conf.example ...
Setting up openssl (1.1.1f-1ubuntu2.2) ...
Setting up bind9-libs:amd64 (1:9.16.1-0ubuntu2.6) ...
Setting up libldap-2.4-2:amd64 (2.4.49+dfsg-2ubuntu1.7) ...
Setting up bind9-host (1:9.16.1-0ubuntu2.6) ...
Setting up bind9-dnsutils (1:9.16.1-0ubuntu2.6) ...
Processing triggers for systemd (245.4-4ubuntu3.4) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Installing curl, cpu-checker, bridge-utils, tmux, ubuntu-fan
Reading package lists...
Building dependency tree...
Reading state information...
curl is already the newest version (7.68.0-1ubuntu2.4).
curl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  msr-tools
The following NEW packages will be installed:
  cpu-checker msr-tools
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 16.9 kB of archives.
After this operation, 66.6 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 msr-tools amd64 1.3-3 [10.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 cpu-checker amd64 0.7-1.1 [6936 B]
Fetched 16.9 kB in 0s (53.0 kB/s)
Selecting previously unselected package msr-tools.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71255 files and directories currently installed.)
Preparing to unpack .../msr-tools_1.3-3_amd64.deb ...
Unpacking msr-tools (1.3-3) ...
Selecting previously unselected package cpu-checker.
Preparing to unpack .../cpu-checker_0.7-1.1_amd64.deb ...
Unpacking cpu-checker (0.7-1.1) ...
Setting up msr-tools (1.3-3) ...
Setting up cpu-checker (0.7-1.1) ...
Processing triggers for man-db (2.9.1-1) ...
Reading package lists...
Building dependency tree...
Reading state information...
bridge-utils is already the newest version (1.6-2ubuntu1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
tmux is already the newest version (3.0a-2ubuntu0.2).
tmux set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  dns-root-data dnsmasq-base libidn11
The following NEW packages will be installed:
  dns-root-data dnsmasq-base libidn11 ubuntu-fan
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 400 kB of archives.
After this operation, 1229 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 dns-root-data all 2019052802 [5300 B]
Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libidn11 amd64 1.33-2.2ubuntu2 [46.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 dnsmasq-base amd64 2.80-1.1ubuntu1.2 [314 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 ubuntu-fan all 0.12.13 [34.5 kB]
Fetched 400 kB in 1s (594 kB/s)
Selecting previously unselected package dns-root-data.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 71269 files and directories currently installed.)
Preparing to unpack .../dns-root-data_2019052802_all.deb ...
Unpacking dns-root-data (2019052802) ...
Selecting previously unselected package libidn11:amd64.
Preparing to unpack .../libidn11_1.33-2.2ubuntu2_amd64.deb ...
Unpacking libidn11:amd64 (1.33-2.2ubuntu2) ...
Selecting previously unselected package dnsmasq-base.
Preparing to unpack .../dnsmasq-base_2.80-1.1ubuntu1.2_amd64.deb ...
Unpacking dnsmasq-base (2.80-1.1ubuntu1.2) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../ubuntu-fan_0.12.13_all.deb ...
Unpacking ubuntu-fan (0.12.13) ...
Setting up dns-root-data (2019052802) ...
Setting up libidn11:amd64 (1.33-2.2ubuntu2) ...
Setting up dnsmasq-base (2.80-1.1ubuntu1.2) ...
Setting up ubuntu-fan (0.12.13) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-fan.service → /lib/systemd/system/ubuntu-fan.service.
Processing triggers for systemd (245.4-4ubuntu3.4) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for dbus (1.12.16-2ubuntu2.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Fetching Juju agent version 2.8.9 for amd64
Attempt 1 to download agent binaries from https://streams.canonical.com/juju/tools/agent/2.8.9/juju-2.8.9-ubuntu-amd64.tgz...
agent binaries from https://streams.canonical.com/juju/tools/agent/2.8.9/juju-2.8.9-ubuntu-amd64.tgz downloaded: HTTP 200; time 44.712562s; size 62255661 bytes; speed 1392370.000 bytes/s Agent binaries downloaded successfully.
4e5d20bdc3b765001537fe2035306d6206c4125b5110bc8c81d41c6e4c75e369 /var/lib/juju/tools/2.8.9-focal-amd64/tools.tar.gz
1091694ab6178e79f62996e70f961acc121eba6bb502ee92116c51bb45d9c4d0 /var/lib/juju/gui/gui.tar.bz2
Installing Juju machine agent
2021-02-24 13:59:39 INFO juju.cmd supercommand.go:54 running jujud [2.8.9 0 a48bedf0ae2096d0cf750b7676da9b0a994cff09 gc go1.14.15]
2021-02-24 13:59:41 INFO juju.agent identity.go:22 writing system identity file
2021-02-24 13:59:41 WARNING juju.mongo mongo.go:479 overwriting args.dataDir (set to /var/lib/juju) to /var/snap/juju-db/common
2021-02-24 13:59:41 INFO juju.mongo mongo.go:484 Ensuring mongo server is running; data directory /var/snap/juju-db/common; port 37017
2021-02-24 13:59:41 WARNING juju.mongo service.go:326 configuring mongod with --noauth flag enabled
2021-02-24 13:59:41 INFO juju.packaging manager.go:103 installing "core" via "snap"
2021-02-24 13:59:41 INFO juju.packaging.manager utils.go:64 Running: snap install core
2021-02-24 13:59:55 INFO juju.packaging manager.go:103 installing "juju-db" via "snap"
2021-02-24 13:59:55 INFO juju.packaging.manager utils.go:64 Running: snap install --channel 4.0/stable juju-db
2021-02-24 14:05:14 INFO juju.worker.peergrouper initiate.go:64 finished InitiateMongoServer
ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
ERROR failed to bootstrap model: subprocess encountered error code 1

Example run:
https://solutions.qa.canonical.com/testruns/testRun/f0cc365f-fa91-4ae1-bc63-1bdd686667eb