Hi Sage! Whoa that was quick. :) On 09/08/2019 16:27, Sage Weil wrote: OK, this is interesting. This is an OpenStack Cinder volumes pool, so all the objects in there belong to RBDs. I couldn't think of any situation in which RBD would create a huge object like that. But, as it happens that PG is currently mapped to a primary OSD that is still on FileStore, so I can do a "find -size +1G" on that mount point, and here's what I get: -rw-r--r-- 1 ceph ceph 4457615932 Mar 29 2018 DIR_3/DIR_9/DIR_6/DIR_C/obj-vS6RN9\uQwvXU9DP__head_DBECC693__2 So, bingo. That's a 4.2GB size file whose size matches that offset exactly. But I'm not familiar with that object name format. How did that object get here? And how do I remove it, considering I seem to be unable to access it? rados -p volumes stat 'obj-vS6RN9\uQwvXU9DP' error stat-ing volumes/obj-vS6RN9\uQwvXU9DP: (2) No such file or directory Or is that file just an artifact that doesn't even map to an object? This is turning out to be a learning experience. :) Thanks again for your help! Cheers, Florian

On 12/08/2019 21:07, Alexandre Marangone wrote: Yes indeed, thank you! Cheers, Florian

Hi Tom, responding back on this briefly so that people are in the loop; I'll have more details in a blog post that I hope to get around to writing. On 12/08/2019 11:34, Thomas Byrne - UKRI STFC wrote: Yes but that doesn't help users much for clusters where very large objects already exist. Even in Luminous, osd_max_object_size defaults to 128M, but if an OSD already has objects larger than that, it will still happily start up and serve data with FileStore — and crash any newly added BlueStore OSDs unfortunate enough to be mapped to a PG with one or more objects that are 4GiB or larger. The pending PR to make this a scrub error even on FileStore OSDs mitigates this issue (https://github.com/ceph/ceph/pull/29579), but it'll still cause a somewhat unexpected surprise for people who have just updated to a version including that fix and suddenly see tons of scrub errors — they would be easily forgiven for assuming they've run into a regression that involves false positives on scrub. "Hey, none of these errors were here before the upgrade, surely there's a problem with the software rather than my data!" We've progressed further in the interim and it appears like I can give all-clears on a couple of concerns that we had: 1. It looks like these objects were not created by an RBD going haywire, but by something actually using librados to create them, presumably long before the cluster ever went into production. 2. I am not changing the subject line so I don't mess up people's list archives if their MUA doesn't correctly thread based on In-Reply-To or References, but it's now evident that this is *not* related to bug #38724 but instead really just due to objects being too large for BlueStore, like Sage said in his first reply. Thanks for the answer — by the way I have been imploring all my colleagues to watch your Cephalocon talk,[1] which was excellent. Cheers, Florian [1] https://youtu.be/niFNZN5EKvE