Fluentd "message does not exist" can happen when emitting logs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla-ansible |
Fix Released
|
Undecided
|
Doug Szumski |
Bug Description
Seen with ElasticSearch output, centralised logging enabled. Example error:
```
"2024-03-14 11:14:36 +0000 [warn]: #0 dump an error event: error_class=
"Payload"=>"fluentd worker is now running worker=0",
"Hostname"
"programname"
"log_level"
"@timestamp"
```
How can this happen?
The Kolla Ansible Fluentd pipeline is not idempotent. For example, this section which is used to standardise the log message field removes the original field:
```
<filter fluent.**>
@type parser
key_name message
format /^(?<Payload>.*)$/
</filter>
```
If a bulk request is sent to Elastic/OpenSearch and not all logs are processed, the default behaviour of Fluentd is to re-emit the unprocessed logs using their original tag. These logs are then re-processed by the Fluentd pipeine. Since the pipeline is not idempotent, this can result in processing errors. In the above example, the 'message' field is no longer present, resulting in the "message does not exist" error.
There are various ways to fix this. Making the pipeline idempotent is not necessary if we use the `retry_tag` in the Elastic/OpenSearch output plugin. Eg. https:/
Changed in kolla-ansible: | |
assignee: | nobody → Doug Szumski (dszumski) |
Fix proposed to branch: master /review. opendev. org/c/openstack /kolla- ansible/ +/917434
Review: https:/