kernels compiled with"CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y" lead to weird clock drifts on some CPUs. This may up ending in DOS

AWS research information added:

We have been investigating this issue. It is important to note that the issue here is not the wall clock, which is the source for gettimeofday(), etc. We're seeing differing behavior in the scheduler clock, which is responsible for process accounting. NTP is not a solution to this problem.

The large jump in the kernel printk() time stamp (as seen in dmesg) is due to the kernel reading the TSC (Time Stamp Counter) from the underlying CPU. The RDTSC instruction is not virtualized, so the counter will not start at zero.

Seeing the printk() timestamps jump around is an indication that the kernel is trusting TSC values too much for the scheduler clock. Most kernels are compiled with the "CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y" configuration. This tells the kernel to mix several sources into the scheduler clock, including the monotonic wall clock and TSC.

Unfortunately, when the kernel detects some newer Intel processor models, it assumes that the hardware time source is stable enough to skip the mixing step. The code that makes the assumption is under recent discussion here.

From arch/x86/kernel/cpu/intel.c:

if (c->x86_power & (1 << 8)) {
  set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);
  set_cpu_cap(c, X86_FEATURE_NONSTOP_TSC);
  if (!check_tsc_unstable())
   sched_clock_stable = 1;

I built a test kernel with the code that forces sched_clock_stable=1 commented out. The time stamps from printk() are stable after the CPU booting phase. With the kernel properly mixing time sources into the scheduler clock, process accounting should be more accurate.

If you are working with a Linux vendor on this problem, point them at this code.

I hope this helps resolve your problem, and apologize that you encountered it while running on EC2.

Own comment:

For example this happens with "Intel(R) Xeon(R) CPU E5506 @ 2.13GHz", but not with "Intel(R) Xeon(R) CPU E5410 @ 2.33GHz". This makes EC2 Autoscaling kind of unreliable :(

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-ec2 2.6.32.311.12
ProcVersionSignature: Ubuntu 2.6.32-308.16-ec2 2.6.32.15+drm33.5
Uname: Linux 2.6.32-308-ec2 x86_64
Architecture: amd64
Date: Wed Mar 23 02:11:07 2011
Ec2AMI: ami-4583b731
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-west-1b
Ec2InstanceType: c1.xlarge
Ec2Kernel: aki-4e794c3a
Ec2Ramdisk: unavailable
ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-meta-ec2