Hi, What limits are there on the "reasonable size" of an rbd? E.g. when I try to create a 1 PB rbd with default 4 MiB objects on my octopus cluster: $ rbd create --size 1P --data-pool rbd.ec rbd.meta/fs 2021-01-20T18:19:35.799+1100 7f47a99253c0 -1 librbd::image::CreateRequest: validate_layout: image size not compatible with object ...which somes from: == src/librbd/image/CreateRequest.cc bool validate_layout(CephContext *cct, uint64_t size, file_layout_t &layout) { if (!librbd::ObjectMap<>::is_compatible(layout, size)) { lderr(cct) << "image size not compatible with object map" << dendl; return false; } == src/librbd/ObjectMap.cc template <typename I> bool ObjectMap<I>::is_compatible(const file_layout_t& layout, uint64_t size) { uint64_t object_count = Striper::get_num_objects(layout, size); return (object_count <= cls::rbd::MAX_OBJECT_MAP_OBJECT_COUNT); } == src/cls/rbd/cls_rbd_types.h static const uint32_t MAX_OBJECT_MAP_OBJECT_COUNT = 256000000; For 4 MiB objects that object count equates to just over 976 TB. Is there any particular reason for that MAX_OBJECT_MAP_OBJECT_COUNT, or it just "this is crazy large, if you're trying to go over this you're doing something wrong, rethink your life..."? Yes, I realise I can increase the size of the objects to get a larger rbd, or drop the object-map support (and the fast-diff that goes along with it). I'm SO glad I found this limit now rather than starting on a smaller rbd and a finding the limit when I tried to grow the rbd underneath a rapidly filling filesystem. What else should I know? Background: I currently have nearly 0.5 PB on XFS (on lvm / raid6) and ZFS that I'm looking to move over to ceph. XFS is a requirement, for the reflinking (sadly not yet available in CephFS: https://tracker.ceph.com/issues/1680). The recommendation for XFS is to start larger, on a thin-provisioned store (hello rbd!), rather than start smaller and grow as needed - e.g. see the thread surrounding: https://www.spinics.net/lists/linux-xfs/msg20099.html Rather than a single large rbd, should I be looking at multiple smaller rbds linked together using lvm or somesuch? What are the tradeoffs? And whilst we're here... for an rbd with the data on an erasure-coded pool, how do you calculate the amount of rbd metadata required if/when the rbd data is fully allocated? Cheers, Chris