Hello Joel, Please be aware that it is not recommended to keep a mix of OSDs created with different bluestore_min_alloc_size values within the same CRUSH device class. The consequence of such a mix is that the balancer will not work properly - instead of evening out the OSD space utilization, it will create a distribution with two bands. This is a bug in the balancer. A ticket has been filed already: https://tracker.ceph.com/issues/64715 On Tue, Mar 12, 2024 at 4:45 AM Joel Davidow <jdavidow@nso.edu> wrote:
For osds that are added new, bfm_bytes_per_block is 4096. However, for osds that were added when the cluster was running octopus, bfm_bytes_per_block remains 65535.
Based on https://github.com/ceph/ceph/blob/1c349451176cc5b4ebfb24b22eaaa754e05cff6c/s... and the space allocation section on page 360 of https://pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf, it appears bfm_bytes_per_block is the bluestore_min_alloc_size that the osd was built with.
Below is a sanitized example of what I was referring to as the osd label (which includes bfm_bytes_per_block) that was run on an osd built under octopus. The cluster was later upgraded to pacific.
user@osd-host:/# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-36 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-36/block": { "osd_uuid": "xxxx", "size": 4000783007744, "btime": "2021-09-14T15:16:55.605860+0000", "description": "main", "bfm_blocks": "61047168", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "65536", "bfm_size": "4000783007744", "bluefs": "1", "ceph_fsid": "xxxx", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "xxxx", "osdspec_affinity": "xxxx", "ready": "ready", "require_osd_release": "16", "whoami": "36" } }
I'm really interested in learning the answers to the questions in the original post.
Thanks, Joel
On Wed, Mar 6, 2024 at 12:11 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Feb 28, 2024, at 17:55, Joel Davidow <jdavidow@nso.edu> wrote:
Current situation ----------------- We have three Ceph clusters that were originally built via cephadm on octopus and later upgraded to pacific. All osds are HDD (will be moving to wal+db on SSD) and were resharded after the upgrade to enable rocksdb sharding.
The value for bluefs_shared_alloc_size has remained unchanged at 65535.
The value for bluestore_min_alloc_size_hdd was 65535 in octopus but is reported as 4096 by ceph daemon osd.<id> config show in pacific.
min_alloc_size is baked into a given OSD when it is created. The central config / runtime value does not affect behavior for existing OSDs. The only way to change it is to destroy / redeploy the OSD.
There was a succession of PRs in the Octopus / Pacific timeframe around default min_alloc_size for HDD and SSD device classes, including IIRC one temporary reversion.
However, the osd label after upgrading to pacific retains the value of 65535 for bfm_bytes_per_block.
OSD label?
I'm not sure if your Pacific release has the back port, but not that along ago `ceph osd metadata` was amended to report the min_alloc_size that a given OSD was built with. If you don't have that, the OSD's startup log should report it.
-- aad
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander E. Patrakov