According to the fluentd-plugin-elasticsearch documentation, the plugin, by default, will only reconnect to the ES cluster when it receives "host unreachable" exception. This can be changed by setting `reconnect_on_error` to True. This is even more strongly recommended for connecting to ES clusters running security guard.
What I'm currently experiencing in my deployment seems to be related: Once fluentd-es plugin loses connectivity to the ES cluster, it never recovers and logs are no longer being sent:
```
2019-05-22 21:47:32 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-05-22 21:47:33 +0000 chunk="58980e875da18f46c6c1030714d07a5d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"monitor.region1.\", :port=>9200, :scheme=>\"https\", :user=>\"logstash\", :password=>\"obfuscated\"}): read timeout reached"
2019-05-23 19:04:44 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-05-23 19:04:45 +0000 chunk="58992c060e9445fe909cb4dadc1751ab" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"monitor.region1.\", :port=>9200, :scheme=>\"https\", :user=>\"logstash\", :password=>\"obfuscated\"}): end of file reached (EOFError)"
2019-05-23 19:04:45 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2019-05-23 19:04:46 +0000 chunk="58992c060e9445fe909cb4dadc1751ab" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"monitor.region1.\", :port=>9200, :scheme=>\"https\", :user=>\"logstash\", :password=>\"obfuscated\"}): end of file reached (EOFError)"
2019-05-23 19:04:46 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2019-05-23 19:04:48 +0000 chunk="58992c060e9445fe909cb4dadc1751ab" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"monitor.region1.\", :port=>9200, :scheme=>\"https\", :user=>\"logstash\", :password=>\"obfuscated\"}): end of file reached (EOFError)"
[...]
```
If I wait enough I can see that fluentd gives up on pushing chunks and drops them.
I'll open a review with a proposed configuration change that I've just deployed on one of my controller nodes to see if it helps.
Fix proposed to branch: master
Review: https:/