Hi Robert, Thanks for pointing that out. This issue stems from version differences between our cluster environments: Our current Pacific cluster (migrating to Reef next week) uses the default configuration: bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P' While Reef and newer versions use: bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32' According to documentation, the bluestore_rocksdb_cfs parameter is only applied during OSD creation (when using --mkfs). Since there was no mention of needing to include L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 in the reshard command, I didn't include these options when resharding the OSDs. Did you? Additionally, min_write_buffer_number_to_merge appears to be a RocksDB option (bluestore_rocksdb_options) with a default value of 1 in Pacific and 6 in Reef. It's unclear whether the effective value is determined by bluestore_rocksdb_options or by the value specified in bluestore_rocksdb_cfs during OSD creation. This probably deserves clarification to prevent inconsistent behavior within the same cluster due to OSDs having different min_write_buffer_number_to_merge values from their creation time. Hopefully Igor can provide some insight on this issue. Regards, Frédéric ----- Le 11 Mar 25, à 12:05, Robert Sander r.sander@heinlein-support.de a écrit :
Hi,
Am 3/11/25 um 11:33 schrieb Frédéric Nass:
$ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard
As a consequence, the 'Object' key/value pairs were stored in the default column family instead of the intended 'O' column family. Resharding again with the correct capital 'O' resolved this issue for the affected overspilling OSDs.
Be aware that there remains an error in both the Quincy and Reef documentation, where the letters 'l' and 'p' should actually be uppercase 'L' and 'P'. Using lowercase 'l' and 'p' wrecks the OSD entirely. I've created this tracker [1] in hopes of getting the Quincy and Reef documentation fixed.
The latest docs seems to have it correct:
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro...
Is there a reason why you omitted "=min_write_buffer_number_to_merge=32" after L and P? These are the default values that I also see on my reef cluster.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io