I see something very similar to this on a three node deploy, in stein and in master. The following task fails: ``` TASK [keystone : Creating admin project, user, role, service, and endpoint] ****************************************************************************************************************************************************************** fatal: [control01]: FAILED! => {"msg": "The conditional check '(keystone_bootstrap.stdout | from_json).changed' failed. The error was: Invalid control character at: line 1 column 284 (char 283)"} ``` If I run the command manually I get the following output: ``` $ docker exec keystone kolla_keystone_bootstrap admin top_secret admin admin http://172.28.128.254:35357 http://172.28.128.254:5000 http://172.28.128.254:5000 RegionOne {"failed": true, "msg": "2019-07-01 12:47:09.993 92 WARNING keystone.access_rules_config.backends.json [-] No config file found for access rules, application credential access rules will be unavailable.: IOError: [Errno 2] No such file or directory: '/etc/keystone/access_rules.json' 2019-07-01 12:47:10.256 92 CRITICAL keystone [-] Unhandled error: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.project' doesn't exist") [SQL: u'INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)'] [parameters: {'is_domain': 1, 'description': 'The default domain', 'extra': '{}', 'enabled': 1, 'domain_id': '<>', 'parent_id': None, 'id': 'default', 'name': 'Default'}] (Background on this error at: http://sqlalche.me/e/f405) 2019-07-01 12:47:10.256 92 ERROR keystone Traceback (most recent call last): 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/bin/keystone-manage", line 10, in 2019-07-01 12:47:10.256 92 ERROR keystone sys.exit(main()) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/manage.py", line 40, in main 2019-07-01 12:47:10.256 92 ERROR keystone cli.main(argv=sys.argv, developer_config_file=developer_config) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 1358, in main 2019-07-01 12:47:10.256 92 ERROR keystone CONF.command.cmd_class.main() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 179, in main 2019-07-01 12:47:10.256 92 ERROR keystone klass.do_bootstrap() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/cli.py", line 170, in do_bootstrap 2019-07-01 12:47:10.256 92 ERROR keystone self.bootstrapper.bootstrap() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/bootstrap.py", line 61, in bootstrap 2019-07-01 12:47:10.256 92 ERROR keystone self._bootstrap_default_domain() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/cmd/bootstrap.py", line 82, in _bootstrap_default_domain 2019-07-01 12:47:10.256 92 ERROR keystone domain=default_domain) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped 2019-07-01 12:47:10.256 92 ERROR keystone __ret_val = __f(*args, **kwargs) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/resource/core.py", line 750, in create_domain 2019-07-01 12:47:10.256 92 ERROR keystone domain_id, project_from_domain, initiator) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped 2019-07-01 12:47:10.256 92 ERROR keystone __ret_val = __f(*args, **kwargs) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/resource/core.py", line 221, in create_project 2019-07-01 12:47:10.256 92 ERROR keystone ret = self.driver.create_project(project_id, project) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/common/sql/core.py", line 516, in wrapper 2019-07-01 12:47:10.256 92 ERROR keystone return method(*args, **kwargs) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/keystone/resource/backends/sql.py", line 235, in create_project 2019-07-01 12:47:10.256 92 ERROR keystone return project_ref.to_dict() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2019-07-01 12:47:10.256 92 ERROR keystone self.gen.next() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1062, in _transaction_scope 2019-07-01 12:47:10.256 92 ERROR keystone yield resource 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2019-07-01 12:47:10.256 92 ERROR keystone self.gen.next() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session 2019-07-01 12:47:10.256 92 ERROR keystone self.session.rollback() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-01 12:47:10.256 92 ERROR keystone self.force_reraise() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-01 12:47:10.256 92 ERROR keystone six.reraise(self.type_, self.value, self.tb) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 664, in _session 2019-07-01 12:47:10.256 92 ERROR keystone self._end_session_transaction(self.session) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 692, in _end_session_transaction 2019-07-01 12:47:10.256 92 ERROR keystone session.commit() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1023, in commit 2019-07-01 12:47:10.256 92 ERROR keystone self.transaction.commit() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in commit 2019-07-01 12:47:10.256 92 ERROR keystone self._prepare_impl() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 466, in _prepare_impl 2019-07-01 12:47:10.256 92 ERROR keystone self.session.flush() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2436, in flush 2019-07-01 12:47:10.256 92 ERROR keystone self._flush(objects) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2574, in _flush 2019-07-01 12:47:10.256 92 ERROR keystone transaction.rollback(_capture_exception=True) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__ 2019-07-01 12:47:10.256 92 ERROR keystone compat.reraise(exc_type, exc_value, exc_tb) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2534, in _flush 2019-07-01 12:47:10.256 92 ERROR keystone flush_context.execute() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute 2019-07-01 12:47:10.256 92 ERROR keystone rec.execute(self) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute 2019-07-01 12:47:10.256 92 ERROR keystone uow, 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj 2019-07-01 12:47:10.256 92 ERROR keystone insert, 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1063, in _emit_insert_statements 2019-07-01 12:47:10.256 92 ERROR keystone c = cached_connections[connection].execute(statement, multiparams) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute 2019-07-01 12:47:10.256 92 ERROR keystone return meth(self, multiparams, params) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection 2019-07-01 12:47:10.256 92 ERROR keystone return connection._execute_clauseelement(self, multiparams, params) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement 2019-07-01 12:47:10.256 92 ERROR keystone distilled_params, 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context 2019-07-01 12:47:10.256 92 ERROR keystone e, statement, parameters, cursor, context 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception 2019-07-01 12:47:10.256 92 ERROR keystone util.raise_from_cause(newraise, exc_info) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause 2019-07-01 12:47:10.256 92 ERROR keystone reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context 2019-07-01 12:47:10.256 92 ERROR keystone cursor, statement, parameters, context 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute 2019-07-01 12:47:10.256 92 ERROR keystone cursor.execute(statement, parameters) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute 2019-07-01 12:47:10.256 92 ERROR keystone result = self._query(query) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query 2019-07-01 12:47:10.256 92 ERROR keystone conn.query(q) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query 2019-07-01 12:47:10.256 92 ERROR keystone self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in _read_query_result 2019-07-01 12:47:10.256 92 ERROR keystone result.read() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read 2019-07-01 12:47:10.256 92 ERROR keystone first_packet = self.connection._read_packet() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet 2019-07-01 12:47:10.256 92 ERROR keystone packet.check_error() 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error 2019-07-01 12:47:10.256 92 ERROR keystone err.raise_mysql_exception(self._data) 2019-07-01 12:47:10.256 92 ERROR keystone File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception 2019-07-01 12:47:10.256 92 ERROR keystone raise errorclass(errno, errval) 2019-07-01 12:47:10.256 92 ERROR keystone ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.project' doesn't exist") [SQL: u'INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)'] [parameters: {'is_domain': 1, 'description': 'The default domain', 'extra': '{}', 'enabled': 1, 'domain_id': '<>', 'parent_id': None, 'id': 'default', 'name': 'Default'}] (Background on this error at: http://sqlalche.me/e/f405) 2019-07-01 12:47:10.256 92 ERROR keystone ", "changed": true} ``` The issue appears to be (at least in this case) that bootstrap isn't working properly. If I manually run the command (which should run as part of bootstrap): `$ docker exec keystone keystone-manage db_sync 2019-07-01 12:49:51.780 101 INFO migrate.versioning.api [-] 66 -> 67... 2019-07-01 12:49:52.683 101 INFO migrate.versioning.api [-] done 2019-07-01 12:49:52.684 101 INFO migrate.versioning.api [-] 67 -> 68... 2019-07-01 12:49:52.694 101 INFO migrate.versioning.api [-] done 2019-07-01 12:49:52.695 101 INFO migrate.versioning.api [-] 68 -> 69... 2019-07-01 12:49:52.705 101 INFO migrate.versioning.api [-] done ` I can run the failing command without error: ``` $ docker exec keystone kolla_keystone_bootstrap admin top_secret admin admin http://172.28.128.254:35357 http://172.28.128.254:5000 http://172.28.128.254:5000 RegionOne {"failed": false, "changed": true} ```