Graphs are empty or showing error "Failed to upgrade legacy queries Datasource named Dashboard1 was not found" in Quincy

Ceph cluster bundle.
It is all on focal, and using quincy/stable channel for ceph charms.

BTW, when deploying with latest/stable channel, which is Octopus in this case, the graphs in ceph-dashboard are shown as expected.

Looks like this change[0] from Pacific is the root cause.
The datasource is now hardcoded to "Dashboard1".
So the workaround is to update the datasource name to "Dashboard1" for now.

[0] https://github.com/ceph/ceph/pull/41633

I've just encountered this bug.

I tried setting telegraf's prometheus_datasource config to Dashboard1 but that didn't seem to do the trick. Looking at the telegraf charm source, it looks like this config option is not used if there is a relation to prometheus.

The process to rename the datasource in Grafana:
- log into grafana as admin
- click the gear icon in the left-hand side
- select Data Sources from the list of config subsections
- click "prometheus - Juju generated source"
- click the name and change it to "Dashboard1"

I couldn't figure out a way to automate that so I went hacking... :)

Admittedly, this is a kludge, but this works to change the hard-coded datasource name:

juju run -a ceph-dashboard -- sudo sed -i.bak \
    -e 's/Dashboard1/prometheus - Juju generated source/g' \
    /usr/share/ceph/mgr/dashboard/frontend/dist/en-US/main.*.js\; \
    sudo systemctl restart ceph-mgr@*

As a warning, the changed file comes directly from the ceph-mgr-dashboard package. Which means this is unlikely to survive a package update.

There is a upstream bug report for this.
https://tracker.ceph.com/issues/53505

Maybe not a smart workaround but by adding a new datasource with the name "Dashboard1" will work as well. I believe with this workaround it will still remain even after charm update or package update.

GRAFANA_IP=$(juju run -m lma-maas --unit grafana/leader \
  "network-get --bind-address website")
PROMETHEUS_IP=$(juju run -m lma-maas --unit prometheus/leader \
  "network-get --bind-address website")
juju run-action -m lma-maas --wait grafana/leader get-login-info --format json | tee grafana-login-info
user="$(cat grafana-login-info | jq -r .[].results.username)"
pass="$(cat grafana-login-info | jq -r .[].results.password)"
auth_base64="$(echo -n ${user}:${pass} | openssl base64)"

curl -k -X POST \
  https://"$GRAFANA_IP":3000/api/datasources \
  -H 'Accept: application/json' \
  -H 'Content-Type:application/json' \
  -H "Authorization: Basic ${auth_base64}" \
  -d @- << EOS
{"name":"Dashboard1","type":"prometheus","url":"http://${PROMETHEUS_IP}:9090","access":"proxy","basicAuth":false}
EOS