heat (kilo) - WARNING keystoneclient.middleware.auth_token [-] This middleware module is deprecated as of v0.10.0 in favor of keystonemiddleware.auth_token

On trusty-kilo, UOSCI's heat cli quick check fails ('heat stack-list' returns non-zero). Inspecting the heat unit logs reveals that the heat-api service is infintely terminating/respawning.

# reproducer
Using bundle: http://bazaar.launchpad.net/~ost-maintainers/openstack-charm-testing/trunk/view/head:/bundles/dev/next-kilo-testing.yaml
juju-deployer -Bv -c next-kilo-testing.yaml -d trusty-kilo
heat stack-list

# end-user symptom, cli output
jenkins@juju-env0-machine-11:~/tools/openstack-charm-testing$ heat stack-list
Unable to establish connection to http://172.17.108.111:8004/v1/3d87e7ae9d254eefabe63cdcf6179ba6/stacks?

# end-user symptom, cli output (debug)
jenkins@juju-env0-machine-11:~/tools/openstack-charm-testing$ heat --debug stack-list
DEBUG (session) REQ: curl -g -i -X GET http://172.17.108.112:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG (session) RESP: [200] date: Thu, 05 Mar 2015 17:22:39 GMT content-type: application/json content-length: 424 vary: X-Auth-Token x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://172.17.108.112:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2) Making authentication request to http://172.17.108.112:5000/v2.0/tokens
DEBUG (session) REQ: curl -g -i -X GET http://172.17.108.111:8004/v1/3d87e7ae9d254eefabe63cdcf6179ba6/stacks? -H "Accept: application/json" -H "User-Agent: python-heatclient" -H "X-Region-Name: RegionOne" -H "X-Auth-Token: {SHA1}b63b83019262ab8d34436633c143a8028861b6f3" -H "Content-Type: application/json" -H "X-Auth-Url: http://172.17.108.112:5000/v2.0"
Traceback (most recent call last):
  File "/usr/local/bin/heat", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main
    HeatShell().main(args)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main
    args.func(client, args)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 581, in do_stack_list
    utils.print_list(stacks, fields, sortby_index=3)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/openstack/common/cliutils.py", line 169, in print_list
    for o in objs:
  File "/usr/local/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 100, in paginate
    stacks = self._list(url, 'stacks')
  File "/usr/local/lib/python2.7/dist-packages/heatclient/openstack/common/apiclient/base.py", line 117, in _list
    body = self.client.get(url).json()
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 292, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 285, in client_request
    resp, body = self.json_request(method, url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request
    raise_exc=False, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 369, in request
    resp = send(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 412, in _send_request
    raise exceptions.ConnectionRefused(msg)
keystoneclient.openstack.common.apiclient.exceptions.ConnectionRefused: Unable to establish connection to http://172.17.108.111:8004/v1/3d87e7ae9d254eefabe63cdcf6179ba6/stacks?

# from heat unit syslog (infinite terminate/respawn of heat-api)
Mar 5 17:32:20 juju-osci-sv08-machine-7 kernel: [50277.184423] init: heat-api main process (26442) terminated with status 1
Mar 5 17:32:20 juju-osci-sv08-machine-7 kernel: [50277.184438] init: heat-api main process ended, respawning
Mar 5 17:32:20 juju-osci-sv08-machine-7 kernel: [50277.868246] init: heat-api main process (26457) terminated with status 1
Mar 5 17:32:20 juju-osci-sv08-machine-7 kernel: [50277.868283] init: heat-api main process ended, respawning
Mar 5 17:32:21 juju-osci-sv08-machine-7 kernel: [50278.592877] init: heat-api main process (26472) terminated with status 1
Mar 5 17:32:21 juju-osci-sv08-machine-7 kernel: [50278.592893] init: heat-api main process ended, respawning

# from heat-api.log (infinite)
2015-03-05 17:35:39.489 30788 WARNING keystoneclient.middleware.auth_token [-] This middleware module is deprecated as of v0.10.0 in favor of keystonemiddleware.auth_token - please update your WSGI pipeline to reference the new middleware package.
2015-03-05 17:35:39.490 30788 WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead.
2015-03-05 17:35:39.494 30788 CRITICAL heat [-] DuplicateOptError: duplicate option: auth_uri
2015-03-05 17:35:39.494 30788 TRACE heat Traceback (most recent call last):
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/bin/heat-api", line 55, in <module>
2015-03-05 17:35:39.494 30788 TRACE heat app = config.load_paste_app()
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/heat/common/config.py", line 321, in load_paste_app
2015-03-05 17:35:39.494 30788 TRACE heat app = wsgi.paste_deploy_app(conf_file, app_name, cfg.CONF)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 883, in paste_deploy_app
2015-03-05 17:35:39.494 30788 TRACE heat return deploy.loadapp("config:%s" % paste_config_file, name=app_name)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2015-03-05 17:35:39.494 30788 TRACE heat return loadobj(APP, uri, name=name, **kw)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2015-03-05 17:35:39.494 30788 TRACE heat return context.create()
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2015-03-05 17:35:39.494 30788 TRACE heat return self.object_type.invoke(self)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke
2015-03-05 17:35:39.494 30788 TRACE heat app = filter(app)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/heat/common/auth_url.py", line 67, in auth_url_filter
2015-03-05 17:35:39.494 30788 TRACE heat return AuthUrlFilter(app, conf)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/heat/common/auth_url.py", line 30, in __init__
2015-03-05 17:35:39.494 30788 TRACE heat self.auth_url = self._get_auth_url()
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/heat/common/auth_url.py", line 38, in _get_auth_url
2015-03-05 17:35:39.494 30788 TRACE heat importutils.import_module(auth_token_module)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo/utils/importutils.py", line 57, in import_module
2015-03-05 17:35:39.494 30788 TRACE heat __import__(import_str)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 355, in <module>
2015-03-05 17:35:39.494 30788 TRACE heat CONF.register_opts(_OPTS, group=_AUTHTOKEN_GROUP)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1679, in __inner
2015-03-05 17:35:39.494 30788 TRACE heat result = f(self, *args, **kwargs)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1836, in register_opts
2015-03-05 17:35:39.494 30788 TRACE heat self.register_opt(opt, group, clear_cache=False)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1683, in __inner
2015-03-05 17:35:39.494 30788 TRACE heat return f(self, *args, **kwargs)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1820, in register_opt
2015-03-05 17:35:39.494 30788 TRACE heat return group._register_opt(opt, cli)
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 1239, in _register_opt
2015-03-05 17:35:39.494 30788 TRACE heat if _is_opt_registered(self._opts, opt):
2015-03-05 17:35:39.494 30788 TRACE heat File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 572, in _is_opt_registered
2015-03-05 17:35:39.494 30788 TRACE heat raise DuplicateOptError(opt.name)
2015-03-05 17:35:39.494 30788 TRACE heat DuplicateOptError: duplicate option: auth_uri
2015-03-05 17:35:39.494 30788 TRACE heat

# juju stat heat
jenkins@juju-env0-machine-11:~/tools/openstack-charm-testing$ juju stat heat
environment: osci-sv08
machines:
  "7":
    agent-state: started
    agent-version: 1.21.3
    dns-name: 172.17.108.111
    instance-id: b1a64fd6-9463-4f99-9ccb-69942cd0d023
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1536M root-disk=10240M
services:
  heat:
    charm: local:trusty/heat-12
    exposed: false
    relations:
      amqp:
      - rabbitmq-server
      identity-service:
      - keystone
      shared-db:
      - mysql
    units:
      heat/0:
        agent-state: started
        agent-version: 1.21.3
        machine: "7"
        open-ports:
        - 8000/tcp
        - 8004/tcp
        public-address: 172.17.108.111