Hi David, On 28/10/2019 20:44, David Monschein wrote:
Hi All,
Running an object storage cluster, originally deployed with Nautilus 14.2.1 and now running 14.2.4.
Last week I was alerted to a new warning from my object storage cluster:
[root@ceph1 ~]# ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details.
I looked into this and found the object and pool in question (default.rgw.log):
[root@ceph1 /var/log/ceph]# grep -R -i 'Large omap object found' . ./ceph.log:2019-10-24 12:21:26.984802 osd.194 (osd.194) 715 : cluster [WRN] Large omap object found. Object: 5:0fbdcb32:usage::usage.17:head Key count: 702330 Size (bytes): 92881228
[root@ceph1 ~]# ceph --format=json pg ls-by-pool default.rgw.log | jq '.[]' | egrep '(pgid|num_large_omap_objects)' | grep -v '"num_large_omap_objects": 0,' | grep -B1 num_large_omap_objects "pgid": "5.70", "num_large_omap_objects": 1, While I was investigating, I noticed an enormous amount of entries in the RGW usage log:
[root@ceph ~]# radosgw-admin usage show | grep -c bucket 223326 [...]
I recently ran into a similar issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/AQNGVY7VJ3K6... You have 702,330 keys on that omap object, so you would have been bitten by the default for osd_deep_scrub_large_omap_object_key_threshold having been revised down from 2,000,000 to 200,000 in 14.2.3: https://github.com/ceph/ceph/commit/d8180c57ac9083f414a23fd393497b2784377735 https://tracker.ceph.com/issues/40583 That's why you didn't see this warning before your recent upgrade.
There are entries for over 223k buckets! This was pretty scary to see, considering we only have maybe 500 legitimate buckets in this fairly new cluster. Almost all of the entries in the usage log are bogus entries from anonymous users. It looks like someone/something was scanning, looking for vulnerabilities, etc. Here are a few example entries, notice none of the operations were successful:
Caveat: whether or not you really *want* to trim the usage log is up to you to decide. If you are suspecting you are dealing with a security breach, you should definitely export and preserve the usage log before you trim it, or else delay trimming until you have properly investigated your problem. *If* you decide you no longer need those usage log entries, you can use "radosgw-admin usage trim" with appropriate --start-date, --end-date, and/or --uid options, to clean them up: https://docs.ceph.com/docs/nautilus/radosgw/admin/#trim-usage Please let me know if that information is helpful. Thank you! Cheers, Florian