mgr crashs in 16.2.5 / clock-skew
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ceph (Ubuntu) |
New
|
Undecided
|
Unassigned |
Bug Description
Hello,
Running inside an KVM, impish with latest ceph version.
Can at least reproduce it in 3 reinstalled fresh ceph clusters.
Heres the crash info for my mgr's:
ceph crash info 2021-09-
{
"archived": "2021-09-13 07:59:37.681606",
"backtrace": [
],
"ceph_version": "16.2.5",
"crash_id": "2021-09-
"entity_name": "mgr.ceph-00002",
"os_id": "21.10",
"os_name": "Ubuntu Impish Indri (development branch)",
"os_version": "21.10 (Impish Indri)",
"os_
"process_name": "ceph-mgr",
"stack_sig": "eccaccb958ebf3
"timestamp": "2021-09-
"utsname_
"utsname_
"utsname_
"utsname_
"utsname_
}
On top MGRs having sometimes "clock-skew" issues, even though the daemon is running its loosing connection and kicked out of the cluster. For sure Host and KVM is ntp synchronized.
Not sure if this "clock-skew" is related to this crash here, but will post the log as soon as i have it again.
Heres are the last lines when the MGR is running, but kicked out of the cluster:
2021-09- 14T00:00: 42.062+ 0000 7f05361b8640 -1 received signal: Hangup from pkill -1 -x ceph-mon| ceph-mgr| ceph-mds| ceph-osd| ceph-fuse| radosgw| rbd-mirror| cephfs- mirror (PID: 1137578) UID: 0 14T00:00: 42.530+ 0000 7f0526ffd640 -1 monclient: _check_ auth_rotating possible clock skew, rotating keys expired way too early (before 2021-09- 13T23:00: 42.534987+ 0000) 14T00:00: 43.530+ 0000 7f0526ffd640 -1 monclient: _check_ auth_rotating possible clock skew, rotating keys expired way too early (before 2021-09- 13T23:00: 43.535155+ 0000)
2021-09-
2021-09-