ironic-conductor stays blocked with "invalid enabled-deploy-interfaces config"

Bug #2039467 reported by Abdullah
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Ironic Conductor Charm
New
Undecided
Unassigned

Bug Description

Testrun https://solutions.qa.canonical.com/testruns/3d640f4a-a151-4020-ae60-2777e1fe7a78/ fails because ironic-conductor stays blocked with:

```
ironic-conductor/0* blocked idle 5/lxd/7 10.246.166.156 invalid enabled-deploy-interfaces config, run set-temp-url-secret action on leader to enable direct deploy method
  filebeat/45 active idle 10.246.166.156 Filebeat ready.
  ironic-conductor-mysql-router/0* active idle 10.246.166.156 Unit is ready
  nrpe/48 active idle 10.246.166.156 5666/tcp icmp Ready
  prometheus-grok-exporter/43 active idle 10.246.166.156 9144/tcp Unit is ready
  telegraf/44 active idle 10.246.166.156 9103/tcp Monitoring ironic-conductor/0 (source version/commit 23.07)
  ubuntu-advantage/42 active idle 10.246.166.156 Attached (esm-apps,esm-infra)
```

While on the outside it looks similar to #1999545, the SQL errors are very different:
```
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service [req-8e645b88-3e96-4862-a262-6af041cf405c - - - - -] Error starting thread.: oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: nodes
[SQL: SELECT nodes.created_at AS nodes_created_at, nodes.updated_at AS nodes_updated_at, nodes.version AS nodes_version, nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.instance_uuid AS nodes_instance_uuid, nodes.name AS nodes_name, nodes.chassis_id AS nodes_chassis_id, nodes.power_state AS nodes_power_state, nodes.target_power_state AS nodes_target_power_state, nodes.provision_state AS nodes_provision_state, nodes.target_provision_state AS nodes_target_provision_state, nodes.provision_updated_at AS nodes_provision_updated_at, nodes.last_error AS nodes_last_error, nodes.instance_info AS nodes_instance_info, nodes.properties AS nodes_properties, nodes.driver AS nodes_driver, nodes.driver_info AS nodes_driver_info, nodes.driver_internal_info AS nodes_driver_internal_info, nodes.clean_step AS nodes_clean_step, nodes.deploy_step AS nodes_deploy_step, nodes.resource_class AS nodes_resource_class, nodes.raid_config AS nodes_raid_config, nodes.target_raid_config AS nodes_target_raid_config, nodes.reservation AS nodes_reservation, nodes.conductor_affinity AS nodes_conductor_affinity, nodes.conductor_group AS nodes_conductor_group, nodes.maintenance AS nodes_maintenance, nodes.maintenance_reason AS nodes_maintenance_reason, nodes.fault AS nodes_fault, nodes.console_enabled AS nodes_console_enabled, nodes.inspection_finished_at AS nodes_inspection_finished_at, nodes.inspection_started_at AS nodes_inspection_started_at, nodes.extra AS nodes_extra, nodes.automated_clean AS nodes_automated_clean, nodes.protected AS nodes_protected, nodes.protected_reason AS nodes_protected_reason, nodes.owner AS nodes_owner, nodes.lessee AS nodes_lessee, nodes.allocation_id AS nodes_allocation_id, nodes.description AS nodes_description, nodes.bios_interface AS nodes_bios_interface, nodes.boot_interface AS nodes_boot_interface, nodes.console_interface AS nodes_console_interface, nodes.deploy_interface AS nodes_deploy_interface, nodes.inspect_interface AS nodes_inspect_interface, nodes.management_interface AS nodes_management_interface, nodes.network_interface AS nodes_network_interface, nodes.raid_interface AS nodes_raid_interface, nodes.rescue_interface AS nodes_rescue_interface, nodes.retired AS nodes_retired, nodes.retired_reason AS nodes_retired_reason, nodes.network_data AS nodes_network_data, nodes.storage_interface AS nodes_storage_interface, nodes.power_interface AS nodes_power_interface, nodes.vendor_interface AS nodes_vendor_interface, nodes.boot_mode AS nodes_boot_mode, nodes.secure_boot AS nodes_secure_boot
FROM nodes
WHERE lower(nodes.reservation) LIKE lower(?) AND nodes.target_power_state IS NOT NULL]
[parameters: ('juju-b89e96-5-lxd-7.nosilo.lab1.solutionsqa',)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service Traceback (most recent call last):
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self.dialect.do_execute(
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service cursor.execute(statement, parameters)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service sqlite3.OperationalError: no such table: nodes
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service The above exception was the direct cause of the following exception:
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service Traceback (most recent call last):
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 806, in run_service
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service service.start()
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/ironic/common/rpc_service.py", line 62, in start
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self._real_start()
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/ironic/common/rpc_service.py", line 74, in _real_start
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self.manager.prepare_host()
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/ironic/conductor/base_manager.py", line 87, in prepare_host
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self.dbapi.clear_node_target_power_state(self.host)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 144, in wrapper
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service with excutils.save_and_reraise_exception() as ectxt:
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self.force_reraise()
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service raise self.value
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 142, in wrapper
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service return f(*args, **kwargs)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/ironic/db/sqlalchemy/api.py", line 1235, in clear_node_target_power_state
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service nodes = [node['uuid'] for node in query]
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2887, in __iter__
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service return self._iter().__iter__()
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2894, in _iter
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service result = self.session.execute(
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 1692, in execute
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service result = conn._execute_20(statement, params or {}, execution_options)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service return meth(self, args_10style, kwargs_10style, execution_options)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service return connection._execute_clauseelement(
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service ret = self._execute_context(
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self._handle_dbapi_exception(
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service raise exception
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service self.dialect.do_execute(
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service cursor.execute(statement, parameters)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service oslo_db.exception.DBNonExistentTable: (sqlite3.OperationalError) no such table: nodes
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service [SQL: SELECT nodes.created_at AS nodes_created_at, nodes.updated_at AS nodes_updated_at, nodes.version AS nodes_version, nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.instance_uuid AS nodes_instance_uuid, nodes.name AS nodes_name, nodes.chassis_id AS nodes_chassis_id, nodes.power_state AS nodes_power_state, nodes.target_power_state AS nodes_target_power_state, nodes.provision_state AS nodes_provision_state, nodes.target_provision_state AS nodes_target_provision_state, nodes.provision_updated_at AS nodes_provision_updated_at, nodes.last_error AS nodes_last_error, nodes.instance_info AS nodes_instance_info, nodes.properties AS nodes_properties, nodes.driver AS nodes_driver, nodes.driver_info AS nodes_driver_info, nodes.driver_internal_info AS nodes_driver_internal_info, nodes.clean_step AS nodes_clean_step, nodes.deploy_step AS nodes_deploy_step, nodes.resource_class AS nodes_resource_class, nodes.raid_config AS nodes_raid_config, nodes.target_raid_config AS nodes_target_raid_config, nodes.reservation AS nodes_reservation, nodes.conductor_affinity AS nodes_conductor_affinity, nodes.conductor_group AS nodes_conductor_group, nodes.maintenance AS nodes_maintenance, nodes.maintenance_reason AS nodes_maintenance_reason, nodes.fault AS nodes_fault, nodes.console_enabled AS nodes_console_enabled, nodes.inspection_finished_at AS nodes_inspection_finished_at, nodes.inspection_started_at AS nodes_inspection_started_at, nodes.extra AS nodes_extra, nodes.automated_clean AS nodes_automated_clean, nodes.protected AS nodes_protected, nodes.protected_reason AS nodes_protected_reason, nodes.owner AS nodes_owner, nodes.lessee AS nodes_lessee, nodes.allocation_id AS nodes_allocation_id, nodes.description AS nodes_description, nodes.bios_interface AS nodes_bios_interface, nodes.boot_interface AS nodes_boot_interface, nodes.console_interface AS nodes_console_interface, nodes.deploy_interface AS nodes_deploy_interface, nodes.inspect_interface AS nodes_inspect_interface, nodes.management_interface AS nodes_management_interface, nodes.network_interface AS nodes_network_interface, nodes.raid_interface AS nodes_raid_interface, nodes.rescue_interface AS nodes_rescue_interface, nodes.retired AS nodes_retired, nodes.retired_reason AS nodes_retired_reason, nodes.network_data AS nodes_network_data, nodes.storage_interface AS nodes_storage_interface, nodes.power_interface AS nodes_power_interface, nodes.vendor_interface AS nodes_vendor_interface, nodes.boot_mode AS nodes_boot_mode, nodes.secure_boot AS nodes_secure_boot
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service FROM nodes
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service WHERE lower(nodes.reservation) LIKE lower(?) AND nodes.target_power_state IS NOT NULL]
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service [parameters: ('juju-b89e96-5-lxd-7.nosilo.lab1.solutionsqa',)]
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service (Background on this error at: https://sqlalche.me/e/14/e3q8)
2023-10-16 06:54:27.715 76070 ERROR oslo_service.service
2023-10-16 07:24:38.977 88988 WARNING oslo_config.cfg [-] Deprecated: Option "sql_connection" from group "database" is deprecated. Use option "connection" from group "database".
2023-10-16 07:24:39.151 88988 WARNING ironic.cmd.conductor [-] With shred_random_overwrite_iterations set to 0 and shred_final_overwrite_with_zeros set to False, disks may NOT be shredded at all, unless they support ATA Secure Erase. This is a possible SECURITY ISSUE!
2023-10-16 07:24:39.611 88988 INFO ironic.common.driver_factory [req-37b8b612-3ea2-4824-89bc-ba64158332b2 - - - - -] Loaded the following hardware types: ['intel-ipmi', 'ipmi']
```

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.