Still trying to solve this one. Here is the corresponding log entry when the large omap object was found: ceph-osd.1284.log.2.gz:2019-09-18 11:43:39.237 7fcd68f96700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 26:86e4c833:::usage.22:head Key count: 2009548 Size (bytes): 369641376 I have since trimmed the entire usage log and disabled it entirely. You can see from the output below that there's nothing in these usage log objects. for i in `rados -p .usage ls`; do echo $i; rados -p .usage listomapkeys $i | wc -l; done usage.29 0 usage.12 0 usage.1 0 usage.26 0 usage.20 0 usage.24 0 usage.16 0 usage.15 0 usage.3 0 usage.19 0 usage.23 0 usage.5 0 usage.11 0 usage.7 0 usage.30 0 usage.18 0 usage.21 0 usage.27 0 usage.13 0 usage.22 0 usage.25 0 . 4 usage.10 0 usage.8 0 usage.9 0 usage.28 0 usage.2 0 usage.4 0 usage.6 0 usage.31 0 usage.17 0 root@infra:~# rados -p .usage listomapkeys usage.22 root@infra:~# On Thu, Sep 19, 2019 at 12:54 PM Charles Alva <charlesalva(a)gmail.com> wrote: > > Could you please share how you trimmed the usage log? > > Kind regards, > > Charles Alva > Sent from Gmail Mobile > > > On Thu, Sep 19, 2019 at 11:46 PM shubjero <shubjero(a)gmail.com> wrote: >> >> Hey all, >> >> Yesterday our cluster went in to HEALTH_WARN due to 1 large omap >> object in the .usage pool (I've posted about this in the past). Last >> time we resolved the issue by trimming the usage log below the alert >> threshold but this time it seems like the alert wont clear even after >> trimming and (this time) disabling the usage log entirely. >> >> ceph health detail >> HEALTH_WARN 1 large omap objects >> LARGE_OMAP_OBJECTS 1 large omap objects >> 1 large objects found in pool '.usage' >> Search the cluster log for 'Large omap object found' for more details. >> >> I've bounced ceph-mon, ceph-mgr, radosgw and even issued osd scrub on >> the two osd's that hold pg's for the .usage pool but the alert wont >> clear. >> >> It's been over 24 hours since I trimmed the usage log. >> >> Any suggestions? >> >> Jared Baker >> Cloud Architect, OICR >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Thanks Casey. I will issue a scrub for the pg that contains this object to speed things along. Will report back when that's done. On Fri, Sep 20, 2019 at 2:50 PM Casey Bodley <cbodley(a)redhat.com> wrote: > > Hi Jared, > > My understanding is that these 'large omap object' warnings are only > issued or cleared during scrub, so I'd expect them to go away the next > time the usage objects get scrubbed. > > On 9/20/19 2:31 PM, shubjero wrote: > > Still trying to solve this one. > > > > Here is the corresponding log entry when the large omap object was found: > > > > ceph-osd.1284.log.2.gz:2019-09-18 11:43:39.237 7fcd68f96700 0 > > log_channel(cluster) log [WRN] : Large omap object found. Object: > > 26:86e4c833:::usage.22:head Key count: 2009548 Size (bytes): 369641376 > > > > I have since trimmed the entire usage log and disabled it entirely. > > You can see from the output below that there's nothing in these usage > > log objects. > > > > for i in `rados -p .usage ls`; do echo $i; rados -p .usage > > listomapkeys $i | wc -l; done > > usage.29 > > 0 > > usage.12 > > 0 > > usage.1 > > 0 > > usage.26 > > 0 > > usage.20 > > 0 > > usage.24 > > 0 > > usage.16 > > 0 > > usage.15 > > 0 > > usage.3 > > 0 > > usage.19 > > 0 > > usage.23 > > 0 > > usage.5 > > 0 > > usage.11 > > 0 > > usage.7 > > 0 > > usage.30 > > 0 > > usage.18 > > 0 > > usage.21 > > 0 > > usage.27 > > 0 > > usage.13 > > 0 > > usage.22 > > 0 > > usage.25 > > 0 > > . > > 4 > > usage.10 > > 0 > > usage.8 > > 0 > > usage.9 > > 0 > > usage.28 > > 0 > > usage.2 > > 0 > > usage.4 > > 0 > > usage.6 > > 0 > > usage.31 > > 0 > > usage.17 > > 0 > > > > > > root@infra:~# rados -p .usage listomapkeys usage.22 > > root@infra:~# > > > > > > On Thu, Sep 19, 2019 at 12:54 PM Charles Alva <charlesalva(a)gmail.com> wrote: > >> Could you please share how you trimmed the usage log? > >> > >> Kind regards, > >> > >> Charles Alva > >> Sent from Gmail Mobile > >> > >> > >> On Thu, Sep 19, 2019 at 11:46 PM shubjero <shubjero(a)gmail.com> wrote: > >>> Hey all, > >>> > >>> Yesterday our cluster went in to HEALTH_WARN due to 1 large omap > >>> object in the .usage pool (I've posted about this in the past). Last > >>> time we resolved the issue by trimming the usage log below the alert > >>> threshold but this time it seems like the alert wont clear even after > >>> trimming and (this time) disabling the usage log entirely. > >>> > >>> ceph health detail > >>> HEALTH_WARN 1 large omap objects > >>> LARGE_OMAP_OBJECTS 1 large omap objects > >>> 1 large objects found in pool '.usage' > >>> Search the cluster log for 'Large omap object found' for more details. > >>> > >>> I've bounced ceph-mon, ceph-mgr, radosgw and even issued osd scrub on > >>> the two osd's that hold pg's for the .usage pool but the alert wont > >>> clear. > >>> > >>> It's been over 24 hours since I trimmed the usage log. > >>> > >>> Any suggestions? > >>> > >>> Jared Baker > >>> Cloud Architect, OICR > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io