tripleo

All OVB jobs in master periodic pipeline fail to install the undercloud while trying to create the ctlplane network with 503 service unavailable on neutron

Bug #1818538 reported by Gabriele Cerami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Gabriele Cerami
tripleo stein-3

Bug Description

This is happening in all OVB job currently.

Example log at

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/7dc4975/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-03-04_11_41_46

Shows:

2019-03-04 11:41:46 | "stderr": [
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,844] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/script < /var/lib/heat-config/deployed/f0d889b9-7280-4073-ac70-315359b3541e.json",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,492] (heat-config) [INFO] |-",
2019-03-04 11:41:46 | " {\"deploy_stdout\": \"ERROR: Network create/update failed.\\n\", \"deploy_stderr\": \"Traceback (most recent call last):\\n File \\\"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\\\", line 281, in <module>\\n network = _ensure_neutron_network(sdk)\\n File \\\"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\\\", line 44, in _ensure_neutron_network\\n network = list(sdk.network.networks(name=CTLPLANE_NETWORK_NAME))\\n File \\\"/usr/lib/python2.7/site-packages/openstack/resource.py\\\", line 1319, in list\\n exceptions.raise_from_response(response)\\n File \\\"/usr/lib/python2.7/site-packages/openstack/exceptions.py\\\", line 229, in raise_from_response\\n http_status=http_status, request_id=request_id\\nopenstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/v2.0/networks?name=ctlplane, No server is available to handle this request.: 503 Service Unavailable\\n\", \"deploy_status_code\": 1}",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,493] (heat-config) [DEBUG] [2019-03-04 11:41:41,879] (heat-config) [INFO] config={\"cloud_name\": \"undercloud\", \"enable_routed_networks\": true, \"home_dir\": \"/home/zuul\", \"local_ip\": \"192.168.24.3\", \"local_subnet\": \"ctlplane-subnet\", \"mtu\": 1350, \"nameservers\": [\"1.1.1.1\", \"1.0.0.1\", \"38.145.33.91\"], \"physical_network\": \"ctlplane\", \"subnets\": {\"ctlplane-subnet\": {\"AllocationPools\": [{\"end\": \"192.168.24.30\", \"start\": \"192.168.24.5\"}], \"DhcpRangeEnd\": \"192.168.24.24\", \"DhcpRangeStart\": \"192.168.24.5\", \"NetworkCidr\": \"192.168.24.0/24\", \"NetworkGateway\": \"192.168.24.1\"}}}",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_server_id=a051f8e5-d82c-4aa2-93f5-117fdc55908f",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_action=CREATE",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_stack_id=undercloud-AllNodesDeploySteps-r3ochb4weckr-UndercloudExtraConfigPost-kvq4indzkukw-UndercloudCtlplaneNetworkDeployment-mk7iorqyyayw-0-g33zaq5ybz25/735a1452-4c83-48e7-b921-e927ffcfa5bd",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_resource_name=TripleOSoftwareDeployment",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_signal_transport=NO_SIGNAL",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,880] (heat-config) [DEBUG] Running /var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,486] (heat-config) [INFO] ERROR: Network create/update failed.",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,486] (heat-config) [DEBUG] Traceback (most recent call last):",
2019-03-04 11:41:46 | " File \"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\", line 281, in <module>",
2019-03-04 11:41:46 | " network = _ensure_neutron_network(sdk)",
2019-03-04 11:41:46 | " File \"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\", line 44, in _ensure_neutron_network",
2019-03-04 11:41:46 | " network = list(sdk.network.networks(name=CTLPLANE_NETWORK_NAME))",
2019-03-04 11:41:46 | " File \"/usr/lib/python2.7/site-packages/openstack/resource.py\", line 1319, in list",
2019-03-04 11:41:46 | " exceptions.raise_from_response(response)",
2019-03-04 11:41:46 | " File \"/usr/lib/python2.7/site-packages/openstack/exceptions.py\", line 229, in raise_from_response",
2019-03-04 11:41:46 | " http_status=http_status, request_id=request_id",
2019-03-04 11:41:46 | "openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/v2.0/networks?name=ctlplane, No server is available to handle this request.: 503 Service Unavailable",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,486] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e. [1]",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,493] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/script",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,493] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/f0d889b9-7280-4073-ac70-315359b3541e.json < /var/lib/heat-config/deployed/f0d889b9-7280-4073-ac70-315359b3541e.notify.json",
2019-03-04 11:41:46 | "[2019-03-04 11:41:44,314] (heat-config) [INFO] ",
2019-03-04 11:41:46 | "[2019-03-04 11:41:44,315] (heat-config) [DEBUG] "

Digging into service logs, neutron server logs for example at

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/7dc4975/logs/undercloud/var/log/containers/neutron/server.log.txt.gz

is full of errors like this

2019-03-04 11:49:22.866 8 CRITICAL neutron [req-e11a4f6a-c966-4aed-9f02-7d48103f49b1 - - - - -] Unhandled error: TypeError: __init__() got an unexpected keyword argument 'set_proctitle'
2019-03-04 11:49:22.866 8 ERROR neutron Traceback (most recent call last):
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module>
2019-03-04 11:49:22.866 8 ERROR neutron sys.exit(main())
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main
2019-03-04 11:49:22.866 8 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 68, in boot_server
2019-03-04 11:49:22.866 8 ERROR neutron server_func()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 23, in eventlet_wsgi_server
2019-03-04 11:49:22.866 8 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 91, in serve_wsgi
2019-03-04 11:49:22.866 8 ERROR neutron LOG.exception('Unrecoverable error: please check log '
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-04 11:49:22.866 8 ERROR neutron self.force_reraise()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-04 11:49:22.866 8 ERROR neutron six.reraise(self.type_, self.value, self.tb)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 88, in serve_wsgi
2019-03-04 11:49:22.866 8 ERROR neutron service.start()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 64, in start
2019-03-04 11:49:22.866 8 ERROR neutron self.wsgi_app = _run_wsgi(self.app_name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 316, in _run_wsgi
2019-03-04 11:49:22.866 8 ERROR neutron app = config.load_paste_app(app_name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app
2019-03-04 11:49:22.866 8 ERROR neutron app = loader.load_app(app_name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2019-03-04 11:49:22.866 8 ERROR neutron return deploy.loadapp("config:%s" % self.config_path, name=name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2019-03-04 11:49:22.866 8 ERROR neutron return loadobj(APP, uri, name=name, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2019-03-04 11:49:22.866 8 ERROR neutron return context.create()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-03-04 11:49:22.866 8 ERROR neutron return self.object_type.invoke(self)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2019-03-04 11:49:22.866 8 ERROR neutron **context.local_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 58, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron reraise(*exc_info)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/compat.py", line 23, in reraise
2019-03-04 11:49:22.866 8 ERROR neutron exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron val = callable(*args, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2019-03-04 11:49:22.866 8 ERROR neutron app = loader.get_app(app_name, global_conf=global_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2019-03-04 11:49:22.866 8 ERROR neutron name=name, global_conf=global_conf).create()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-03-04 11:49:22.866 8 ERROR neutron return self.object_type.invoke(self)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2019-03-04 11:49:22.866 8 ERROR neutron **context.local_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 58, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron reraise(*exc_info)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/compat.py", line 23, in reraise
2019-03-04 11:49:22.866 8 ERROR neutron exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron val = callable(*args, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 47, in pipeline_factory
2019-03-04 11:49:22.866 8 ERROR neutron app = loader.get_app(pipeline[-1])
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2019-03-04 11:49:22.866 8 ERROR neutron name=name, global_conf=global_conf).create()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-03-04 11:49:22.866 8 ERROR neutron return self.object_type.invoke(self)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2019-03-04 11:49:22.866 8 ERROR neutron return fix_call(context.object, context.global_conf, **context.local_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 58, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron reraise(*exc_info)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/compat.py", line 23, in reraise
2019-03-04 11:49:22.866 8 ERROR neutron exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron val = callable(*args, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 25, in _factory
2019-03-04 11:49:22.866 8 ERROR neutron return pecan_app.v2_factory(global_config, **local_config)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory
2019-03-04 11:49:22.866 8 ERROR neutron startup.initialize_all()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
2019-03-04 11:49:22.866 8 ERROR neutron manager.init()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 314, in init
2019-03-04 11:49:22.866 8 ERROR neutron NeutronManager.get_instance()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 265, in get_instance
2019-03-04 11:49:22.866 8 ERROR neutron cls._create_instance()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-03-04 11:49:22.866 8 ERROR neutron return f(*args, **kwargs)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 251, in _create_instance
2019-03-04 11:49:22.866 8 ERROR neutron cls._instance = cls()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 136, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron plugin_provider)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 175, in _get_plugin_instance
2019-03-04 11:49:22.866 8 ERROR neutron plugin_inst = plugin_class()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 128, in wrapper
2019-03-04 11:49:22.866 8 ERROR neutron return f(*args, **kwargs)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 240, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron self._setup_dhcp()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 264, in _setup_dhcp
2019-03-04 11:49:22.866 8 ERROR neutron self.add_periodic_dhcp_agent_status_check()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 193, in add_periodic_dhcp_agent_status_check
2019-03-04 11:49:22.866 8 ERROR neutron self.remove_networks_from_down_agents
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 94, in add_agent_status_check_worker
2019-03-04 11:49:22.866 8 ERROR neutron initial_delay)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/worker.py", line 35, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron super(PeriodicWorker, self).__init__(worker_process_count=0)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/worker.py", line 24, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron set_proctitle=set_proctitle
2019-03-04 11:49:22.866 8 ERROR neutron TypeError: __init__() got an unexpected keyword argument 'set_proctitle'
2019-03-04 11:49:22.866 8 ERROR neutron

Revision history for this message
Gabriele Cerami (gcerami) wrote :

Looks like this has been caused by this change

https://review.openstack.org/637019

not sure how this started to affect TripleO though

summary: All OVB jobs in master periodic pipeline fail to install the undercloud
- while trying to create the ctlplane networ with 503 service unavailable
+ while trying to create the ctlplane network with 503 service unavailable
+ on neutron
Revision history for this message
Alex Schultz (alex-schultz) wrote :

We need neutron-lib 1.25.0. It added https://review.openstack.org/#/c/637119/ which was required for https://review.openstack.org/#/c/637019/

Revision history for this message
Gabriele Cerami (gcerami) wrote :

https://review.rdoproject.org/r/19100 has been proposed and merged to bump neutron-lib release

Changed in tripleo:
status: Triaged → Fix Committed
Revision history for this message
Gabriele Cerami (gcerami) wrote :

still happening with the latest periodic run

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/88df38a/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2019-03-05_14_43_23_842

I can see neutron-lib package is 1.25 here

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/88df38a/logs/undercloud/var/log/extra/rpm-list.txt.gz

So bumping didn't fix the error

Revision history for this message
Gabriele Cerami (gcerami) wrote :

Jobs are now passing. No additional fixes required.

Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.