Updating terminated TLS listener may leak sensitive data in logs for task_flow

Bug #2028546 reported by Daniel Failing
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
octavia
New
Undecided
Unassigned

Bug Description

When creating a terminated TLS listener, the certificate, private key and passphrase might be leaked to the worker.log

We found those information as the taskflow was transitioning from RUNNING to FAILURE and the atoms were logged with

2023-07-24 14:00:43.380 1784274 WARNING octavia.common.base_taskflow [ ] Task 'octavia.controller.worker.v2.tasks.amphora_driver_tasks.ListenersUpdate' ([...]) transitioned into state 'FAILURE' from state 'RUNNING'
3 predecessors (most recent first):
  Atom 'octavia.controller.worker.v2.tasks.database_tasks.MarkHealthMonitorPendingCreateInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'health_mon': {'admin_state_up': True, 'delay': 3, 'expected_codes': '200', 'healthmonitor_id': '[...]', 'http_method': 'GET', 'max_retries': 3, 'max_retries_down': 3, 'name': '[...]', 'pool_id': '[...]', 'timeout': 3, 'type': 'HTTPS', 'url_path': '/healthcheck', 'http_version': 1.0, 'domain_name': None, 'project_id': '[...]'}}, 'provides': None}
  |__Atom 'octavia.controller.worker.v2.tasks.lifecycle_tasks.HealthMonitorToErrorOnRevertTask' [...]

Leaked data in this atom and below, e.g. all information in default_tls_container_data should be considered confidential

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

Hi,

We did a patch for similar issues https://review.opendev.org/c/openstack/octavia/+/871468
but it covers only non-ERROR/non-WARNING messages, it replaces the value of the private elements in a dict with '***'

For this particular case, it's a dump of the flow and its atoms by taskflow, it's more complicated as the strings are directly formatted in taskflow, there's no way to change the value of the dict.

Revision history for this message
Michael Johnson (johnsom) wrote :

I would take a similar approach to how other keys are passed through taskflow, encrypt them before they enter and decrypt as the leave.
See: https://github.com/openstack/octavia/blob/master/octavia/certificates/common/local.py#L54
and
https://github.com/openstack/octavia/blob/master/octavia/controller/worker/v2/tasks/amphora_driver_tasks.py#L402

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

Another approach (discussed during the PTG) would be to disable the "tree" output from taskflow, it may be useful for troubleshooting (maybe in DEBUG?), but in most cases, the backtrace raised by the task is enough.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.