Improvements to the default configuration of Kafka

One may also want to update the log flush policy to enable it by default

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
# DEFAULT: Commented
log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
# Default: Commented.
log.flush.interval.ms=1000

Not sure I understand the tag "not-a-charm". Can you explain? This is a bug
in the charm to deploy Kafka, not a bug in Kafka.

Thx,
Samuel

Best,
Samuel

--
Samuel Cozannet
Cloud, Big Data and IoT Strategy Team
Strategic Program Manager
Changing the Future of Cloud
Ubuntu <http://ubuntu.com> / Canonical <http://canonical.com> UK LTD
<email address hidden>
+33 616 702 389

On Mon, Dec 8, 2014 at 4:56 PM, Whit Morriss <email address hidden>
wrote:

> ** Tags added: not-a-charm
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1396610
>
> Title:
> Improvements to the default configuration of Kafka
>
> Status in Juju Charms:
> In Progress
>
> Bug description:
> By default, Kafka will not enable the log cleaner. This means that in
> a production environment the log output from Kafka will consume all
> storage space after a while.
>
> This while depends very much on the workload Kafka is used for. If it
> is fast paced (as we expect from Kafka) this can be a few hours.
>
> From a "security" / "charm acceptance" PoV, a secure opinionated
> deployment may use the below settings for
> /opt/kafka/server.properties:
>
>
> # The minimum age of a log file to be eligible for deletion
> # DEFAULT: log.retention.hours=168
> log.retention.hours=24
>
> # A size-based retention policy for logs. Segments are pruned from the
> log as long as the remaining
> # segments don't drop below log.retention.bytes.
> # DEFAULT: log.retention.bytes=1073741824
> log.retention.bytes=33554432
>
>
> # The maximum size of a log segment file. When this size is reached a
> new log segment will be created.
> # DEFAULT: log.segment.bytes=536870912
> log.segment.bytes=8388608
>
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=60000
>
>
> # By default the log cleaner is disabled and the log retention policy
> will default to just delete segments after their retention expires.
> # If log.cleaner.enable=true is set the cleaner will be enabled and
> individual logs can then be marked for log compaction.
> # DEFAULT : log.cleaner.enable=false
> log.cleaner.enable=true
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charms/+bug/1396610/+subscriptions
>

I've rewritten an apache-kafka charm, and this will likely apply. I'm taking ownership.