Three OSDs, holding the 3 replicas of a PG here are only half-starting, and hence that single PG gets stuck as "stale+active+clean". All died of suicide timeout while walking over a huge omap (pool 7 'default.rgw.buckets.index') and would not get the PG 7.b back online again. From the logs, they try to start normally, get into a bit of leveldb things, play the journal and then say nothing more. 2019-11-19 15:15:46.967543 7fe644fad840 0 set uid:gid to 167:167 (ceph:ceph) 2019-11-19 15:15:46.967600 7fe644fad840 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-osd, pid 5149 2019-11-19 15:15:47.026065 7fe644fad840 0 pidfile_write: ignore empty --pid-file 2019-11-19 15:15:47.078291 7fe644fad840 0 filestore(/var/lib/ceph/osd/ceph-22) backend xfs (magic 0x58465342) 2019-11-19 15:15:47.079317 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2019-11-19 15:15:47.079331 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2019-11-19 15:15:47.079352 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: splice is supported 2019-11-19 15:15:47.080287 7fe644fad840 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2019-11-19 15:15:47.080529 7fe644fad840 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-22) detect_feature: extsize is disabled by conf 2019-11-19 15:15:47.095819 7fe644fad840 1 leveldb: Recovering log #2731809 2019-11-19 15:15:47.119792 7fe644fad840 1 leveldb: Level-0 table #2731812: started 2019-11-19 15:15:47.132107 7fe644fad840 1 leveldb: Level-0 table #2731812: 140642 bytes OK 2019-11-19 15:15:47.143782 7fe644fad840 1 leveldb: Delete type=0 #2731809 2019-11-19 15:15:47.147198 7fe644fad840 1 leveldb: Delete type=3 #2731792 2019-11-19 15:15:47.159339 7fe644fad840 0 filestore(/var/lib/ceph/osd/ceph-22) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2019-11-19 15:15:47.243262 7fe644fad840 1 journal _open /var/lib/ceph/osd/ceph-22/journal fd 18: 21472739328 bytes, block size 4096 bytes, directio = 1, aio = 1 At this point they consume a ton of cpu, systemd thinks all is fine, and this has been going on for some 5 hours. ceph -s think they are down, I can't talk to the OSDs remotely from a mon, but ceph daemon on the OSD hosts works normally, except I can't do anything from there except get conf or perf numbers. Strace shows they all keep looping over the same sequence: machine1: stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7fffd7c98080) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7fffd7c98080) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-270/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 machine2: stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffe0b664240) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffe0b664240) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-243/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 machine3: stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffc63518650) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/DIR_D", 0x7ffc63518650) = -1 ENOENT (No such file or directory) stat("/var/lib/ceph/osd/ceph-22/current/7.b_head/DIR_B/DIR_4/\\.dir.31716e6b-28c9-42e6-81ed-d27e3b714a9c.47687923.1711__head_6D57DD4B__7", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 Help wanted. -- May the most significant bit of your life be positive.