Dear All, My ceph cluster is giving Large OMAP warning from approx 2-3 Months. I tried a few things like : *Deep scrub of PGs* *Compact OSDs* *Trim log* But these didn't work out. I guess the main issue is that 4 shards in replication site are always recovering from 2-3 months. Any suggestions are highly appreciated. Sync status: root@drhost1:~# radosgw-admin sync status realm e259e0a92 (object-storage) zonegroup 7a8606d2 (staas) zone c8022ad1 (repstaas) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 2072ee2d32 (masterstaas) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [7,90,100] oldest incremental change not applied: 2025-03-12T13:14:10.268469+0530 [7] 4 shards are recovering recovering shards: [31,41,55,80] Master site: 1. *root@master1:~# for obj in $(rados ls -p masterstaas.rgw.log); do echo "$(rados listomapkeys -p masterstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head -10* 1225387 data_log.91 1225065 data_log.86 1224662 data_log.87 1224448 data_log.92 1224018 data_log.89 1222156 data_log.93 1201489 data_log.83 1174125 data_log.90 363498 data_log.84 258709 data_log.6 2. *root@master1:~# for obj in data_log.91 data_log.86 data_log.87 data_log.92 data_log.89 data_log.93 data_log.83 data_log.90; do rados stat -p masterstaas.rgw.log $obj; done* masterstaas.rgw.log/data_log.91 mtime 2025-02-24T15:09:25.000000+0530, size 0 masterstaas.rgw.log/data_log.86 mtime 2025-02-24T15:01:25.000000+0530, size 0 masterstaas.rgw.log/data_log.87 mtime 2025-02-24T15:02:25.000000+0530, size 0 masterstaas.rgw.log/data_log.92 mtime 2025-02-24T15:11:01.000000+0530, size 0 masterstaas.rgw.log/data_log.89 mtime 2025-02-24T14:54:55.000000+0530, size 0 masterstaas.rgw.log/data_log.93 mtime 2025-02-24T14:53:25.000000+0530, size 0 masterstaas.rgw.log/data_log.83 mtime 2025-02-24T14:16:21.000000+0530, size 0 masterstaas.rgw.log/data_log.90 mtime 2025-02-24T15:05:25.000000+0530, size 0 *3. ceph cluster log :* 2025-02-22T04:18:27.324886+0530 osd.173 (osd.173) 19 : cluster [WRN] Large omap object found. Object: 124:b2ddf551:::data_log.93:head PG: 124.8aafbb4d (124.d) Key count: 1218170 Size (bytes): 297085860 2025-02-22T04:18:28.735886+0530 osd.65 (osd.65) 308 : cluster [WRN] Large omap object found. Object: 124:f2081d70:::data_log.92:head PG: 124.eb8104f (124.f) Key count: 1220420 Size (bytes): 295240028 2025-02-22T04:18:30.668884+0530 mon.master1 (mon.0) 7974038 : cluster [WRN] Health check update: 3 large omap objects (LARGE_OMAP_OBJECTS) 2025-02-22T04:18:31.127585+0530 osd.18 (osd.18) 224 : cluster [WRN] Large omap object found. Object: 124:d1061236:::data_log.86:head PG: 124.6c48608b (124.b) Key count: 1221047 Size (bytes): 295398274 2025-02-22T04:18:33.189561+0530 osd.37 (osd.37) 32665 : cluster [WRN] Large omap object found. Object: 124:9a2e04b7:::data_log.87:head PG: 124.ed207459 (124.19) Key count: 1220584 Size (bytes): 295290366 2025-02-22T04:18:35.007117+0530 osd.77 (osd.77) 135 : cluster [WRN] Large omap object found. Object: 124:6b9e929a:::data_log.89:head PG: 124.594979d6 (124.16) Key count: 1219913 Size (bytes): 295127816 2025-02-22T04:18:36.189141+0530 mon.master1 (mon.0) 7974039 : cluster [WRN] Health check update: 5 large omap objects (LARGE_OMAP_OBJECTS) 2025-02-22T04:18:36.340247+0530 osd.112 (osd.112) 259 : cluster [WRN] Large omap object found. Object: 124:0958bece:::data_log.83:head PG: 124.737d1a90 (124.10) Key count: 1200406 Size (bytes): 290280292 2025-02-22T04:18:38.523766+0530 osd.73 (osd.73) 1064 : cluster [WRN] Large omap object found. Object: 124:fddd971f:::data_log.91:head PG: 124.f8e9bbbf (124.3f) Key count: 1221183 Size (bytes): 295425320 2025-02-22T04:18:42.619926+0530 osd.92 (osd.92) 285 : cluster [WRN] Large omap object found. Object: 124:7dc404fa:::data_log.90:head PG: 124.5f2023be (124.3e) Key count: 1169895 Size (bytes): 283025576 2025-02-22T04:18:44.242655+0530 mon.master1 (mon.0) 7974043 : cluster [WRN] Health check update: 8 large omap objects (LARGE_OMAP_OBJECTS) Replica site: 1. *for obj in $(rados ls -p repstaas.rgw.log); do echo "$(rados listomapkeys -p repstaas.rgw.log $obj | wc -l) $obj";done | sort -nr | head -10* 432850 data_log.91 432384 data_log.87 432323 data_log.93 431783 data_log.86 431510 data_log.92 427959 data_log.89 414522 data_log.90 407571 data_log.83 151015 data_log.84 109790 data_log.4 2. *ceph cluster log:* grep -ir "Large omap object found" /var/log/ceph/ /var/log/ceph/ceph-mon.drhost1.log:2025-03-12T14:49:59.997+0530 7fc4ad544700 0 log_channel(cluster) log [WRN] : Search the cluster log for 'Large omap object found' for more details. /var/log/ceph/ceph.log:2025-03-12T14:49:02.078108+0530 osd.10 (osd.10) 21 : cluster [WRN] Large omap object found. Object: 6:b2ddf551:::data_log.93:head PG: 6.8aafbb4d (6.d) Key count: 432323 Size (bytes): 105505884 /var/log/ceph/ceph.log:2025-03-12T14:49:02.389288+0530 osd.48 (osd.48) 37 : cluster [WRN] Large omap object found. Object: 6:d1061236:::data_log.86:head PG: 6.6c48608b (6.b) Key count: 431782 Size (bytes): 104564674 /var/log/ceph/ceph.log:2025-03-12T14:49:07.166954+0530 osd.24 (osd.24) 24 : cluster [WRN] Large omap object found. Object: 6:0958bece:::data_log.83:head PG: 6.737d1a90 (6.10) Key count: 407571 Size (bytes): 98635522 /var/log/ceph/ceph.log:2025-03-12T14:49:09.100110+0530 osd.63 (osd.63) 5 : cluster [WRN] Large omap object found. Object: 6:9a2e04b7:::data_log.87:head PG: 6.ed207459 (6.19) Key count: 432384 Size (bytes): 104712350 /var/log/ceph/ceph.log:2025-03-12T14:49:08.703760+0530 osd.59 (osd.59) 11 : cluster [WRN] Large omap object found. Object: 6:6b9e929a:::data_log.89:head PG: 6.594979d6 (6.16) Key count: 427959 Size (bytes): 103773777 /var/log/ceph/ceph.log:2025-03-12T14:49:11.126132+0530 osd.40 (osd.40) 24 : cluster [WRN] Large omap object found. Object: 6:f2081d70:::data_log.92:head PG: 6.eb8104f (6.f) Key count: 431508 Size (bytes): 104520406 /var/log/ceph/ceph.log:2025-03-12T14:49:13.799473+0530 osd.43 (osd.43) 61 : cluster [WRN] Large omap object found. Object: 6:fddd971f:::data_log.91:head PG: 6.f8e9bbbf (6.1f) Key count: 432850 Size (bytes): 104418869 /var/log/ceph/ceph.log:2025-03-12T14:49:14.398480+0530 osd.3 (osd.3) 55 : cluster [WRN] Large omap object found. Object: 6:7dc404fa:::data_log.90:head PG: 6.5f2023be (6.1e) Key count: 414521 Size (bytes): 100396561 /var/log/ceph/ceph.log:2025-03-12T14:50:00.000484+0530 mon.drhost1 (mon.0) 207423 : cluster [WRN] Search the cluster log for 'Large omap object found' for more details. Regards, Danish