senlin client does not appear to respect OS_CACERT env var

I have an SSL-encrypted keystone api endpoint. (In fact, I run all OpenStack services with end-to-end SSL encryption -- there are no proxies or SSL termination.) I use a self-signed SSL cert, which should be no problem as long as I supply my public CA certificate as part of the OS_CACERT environment variable in my keystonerc/openrc file, and also specify it in all the [keystone_authtoken] section of all my servcies, which I do. Every single service in my cloud works fine this way (keystone, glance, cinder, nova, neutron, horizon, rabbitmq, barbican -- all works), except for Senlin.

In CentOS 7, '/etc/pki/tls/certs/ca-bundle.crt' is a symlink to '/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem':

# ls -la /etc/pki/tls/certs/ca-bundle.crt
lrwxrwxrwx. 1 root root 49 Jan 12 16:59 /etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

I find that the senlinclient requires my CA cert to be appended to this file, or else senlinclient does not work, regardless of the OS_CACERT env variable. I'll do my best to demonstrate.

# ls -latr tls-ca-bundle.pem*
-r--r--r--. 1 root root 260022 Jan 18 00:40 tls-ca-bundle.pem.works
-r--r--r--. 1 root root 258090 Jan 18 11:23 tls-ca-bundle.pem

'tls-ca-bundle.pem.works' is the same file as 'tls-ca-bundle.pem', only with my self-signed CA cert, base64-encoded PEM-sytle, appended to the end of the file. (I will refrain from sharing my actual CA cert because I work for a Federal agency.)

I'll now source my keystonerc file and try to run a basic senlin command:

[root@localhost pem]# . /root/keystonerc_admin
[root@localhost pem]# printenv |grep OS_CACERT
OS_CACERT=/etc/openldap/cacerts/adapt_2_root_ca.pem
[root@localhost pem]# openstack --debug cluster list

See full output: https://pastebin.com/Bnqyk71i

Okay, now I'll change tls-ca-bundle.pem to the version containing my SSL CA cert:

[root@localhost pem]# cp -p tls-ca-bundle.pem tls-ca-bundle.pem.old
[root@localhost pem]# ln -s --force tls-ca-bundle.pem.works tls-ca-bundle.pem
[root@localhost pem]# ls -la tls-ca-bundle.pem*
lrwxrwxrwx. 1 root root 23 Jan 18 11:33 tls-ca-bundle.pem -> tls-ca-bundle.pem.works
-r--r--r--. 1 root root 258090 Jan 18 11:23 tls-ca-bundle.pem.old
-r--r--r--. 1 root root 260022 Jan 18 00:40 tls-ca-bundle.pem.works
[root@localhost pem]#

Now I'll rerun the same command:

# openstack --debug cluster list

See full output: https://pastebin.com/9n9VzMMZ

You'll note that, this time, the client doesn't fail -- it gets a token. However, the command still fails because Senlin API has the same problem, and can't talk to SSL either.

In general, there is some similar issue with the way Senlin is handling SSL CA certs. I've found a work-around for the client, but it's still a bug. I have not yet found a work-around for the API