Hi folks, CephFS has some oddities around space reporting that have gotten noisier over time, and I’d like to take a stab at improving the situation: 1. The MDS (and consequently, reporting in general) has no idea how much space is actually allocated for a file. (We report the size, but that is just the last byte written — it could be one byte at offset 1TB, or it could be a whole terabyte of data written — we’ve no idea.) 2. The MDS has no idea how much space a snapshot is using up (and neither does any other global operator). This situation generally arises because the clients do all IO directly to the OSDs, and don’t necessarily themselves know what their ops are doing to a file — it might be an overwrite, or it could be extending the file. Write operations classically could only return a success or error code, so any system to change that would require extra-OSD coordination (yuck). However, in 2019 we added the ability for write operations to return a limited amount of data in https://github.com/ceph/ceph/pull/30581. This is generally expected to be object class ops working on a custom data structure, and I believe RGW is making use of the functionality for some of its indices. So I’d like to propose a new write operation that takes advantage of returning information, to let CephFS efficiently track space allocations. The discussion part of this is not so much about how to implement it, but the bounds that are safe: right now, the limits on write returns are surprisingly generous — up to 64 bytes *per op* in the write. But I don’t think we push these boundaries, and as they are only used by index operations, most writes in a cluster do not return any data at all. If we implement this new CEPH_OSD_OP_WRITE_ALLOC_RETURN operation, we can expect most writes to be storing extra return data in the pglog. There are three possibilities for what this op could return, each more useful than the last: 1. Return the newly-written range for the operation as a simple integer. This is just 32 bits. (I could convinced to define it even smaller, but as we allow writes up to ~90MB by default it should probably handle that large a number — which requires 27 bits.) 2. Return the newly-written range for the operation, the amount of space which overwrote a previous snapshot, and the newest snapid which was overwritten. This is 32 bits for the newly-written range, plus 64 bits for the snapid and 32 bits for the overlap range (ie, 96 bits extra over option 1). 3. Return the newly-written range for the operation, and a full diff of the clone_overlap for the object generated by this operation. This would be 32 bits for the newly-written range, and then an encoded map of pairs (so I believe 32 bits for the size of the map, and then 96 bits per entry) (3) is definitely the most useful — it lets us fully and accurately track the space used for an individual file, and for CephFS snapshots, in real time. The concern is that it is not very bounded in size, since that will be determined by the snapshot schedule and overwrite churn (though I do expect it to be small in most cases). (2) at least lets us guess about the size of snapshots, but unfortunately means that while we can accurately report the total space used on snapshots for a particular file/subvolume, we can’t attribute that space to a particular snapshot under some very common patterns (ie, if you snapshot hourly and keep 12 hourly snapshots, a daily, and a weekly snapshot). (1) Lets us accurately track HEAD size but doesn’t do anything at all for snapshots. I am trying to see if I can find any telemetry or metrics on how many snapshots people actually maintain on a per-file basis, or how many are likely to be impacted by a particular operation, but I don’t have those numbers yet (if they’re even obtainable). So my question is: Can I implement (3), or is that too scary? Is there some set of bounds I could provide beyond which it returns ETOOBIG and has to be manually queried with LIST_SNAPS? Thanks! -Greg