Bluestore - How to review config?
Hello, Sorry if this has been asked before... A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance. While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware. I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration. Could anybody point me to a command or link to documentation on this? Thanks. -Dave -- Dave Hall Binghamton University
DOCUMENTATION TAG (Sorry, Dave, I don't have an answer. I just have the power to fix the documentation when there is an answer.) On Tue, May 5, 2020 at 9:53 AM Dave Hall <kdhall@binghamton.edu> wrote:
Hello,
Sorry if this has been asked before...
A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance.
While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware.
I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration.
Could anybody point me to a command or link to documentation on this?
Thanks.
-Dave
-- Dave Hall Binghamton University _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dave, Probably not complete but I know 2 interesting ways to get configuration of a Bluestore OSD: 1/ the /show-label/ option of /ceph-bluestore-tool/ command Ex: $ sudo ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0/ 2/ the /config show/ and /perf dump/ parameters of the OSD /admin-daemon/ option Ex: $ sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show and $ sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok perf dump The last both ones are really verbose, but you can grep for a specific information... Indeed, it could be really useful to have those kinds of information on the Ceph dashboard :) For example, the space used by the db (in my case 2 out of 80 GB on a separate nvme WAL/DB): $ sudo ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok perf dump |grep db "db_total_bytes": 80015777792, "db_used_bytes": 2092949504, "max_bytes_db": 2092949504, ... Hope this helps! Regards, Hervé On 05/05/2020 01:52, Dave Hall wrote:
Hello,
Sorry if this has been asked before...
A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance.
While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware.
I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration.
Could anybody point me to a command or link to documentation on this?
Thanks.
-Dave
Hi Dave, wouldn't this help (particularly "Viewing runtime settings" section): https://docs.ceph.com/docs/nautilus/rados/configuration/ceph-conf/ Thanks, Igor On 5/5/2020 2:52 AM, Dave Hall wrote:
Hello,
Sorry if this has been asked before...
A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance.
While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware.
I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration.
Could anybody point me to a command or link to documentation on this?
Thanks.
-Dave
Is there a way to get the block,block.db,block.wal path and size? what if all of them or some of them are colocated in one disk? I can get the info from a wal,db,block colocated osd like below: ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0/ { "/var/lib/ceph/osd/ceph-0//block": { "osd_uuid": "199f2445-af9e-4172-8231-6d98858684e8", "size": 107268255744, "btime": "2020-02-15 21:53:42.972004", "description": "main", "bluefs": "1", "ceph_fsid": "83a73817-3566-4044-91b6-22cee6753515", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "ready": "ready", "require_osd_release": "\u000c", "whoami": "0" } } but there is no path of the block. ceph osd metadata 0 { "id": 0, "arch": "x86_64", "back_addr": "172.18.2.178:6801/12605", "back_iface": "ens160", "bluefs": "1", "bluefs_db_access_mode": "blk", "bluefs_db_block_size": "4096", "bluefs_db_dev": "8:16", "bluefs_db_dev_node": "sdb", "bluefs_db_driver": "KernelDevice", "bluefs_db_model": "Virtual disk ", "bluefs_db_partition_path": "/dev/sdb2", "bluefs_db_rotational": "1", "bluefs_db_size": "107268255744", "bluefs_db_type": "hdd", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev": "8:16", "bluestore_bdev_dev_node": "sdb", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_model": "Virtual disk ", "bluestore_bdev_partition_path": "/dev/sdb2", "bluestore_bdev_rotational": "1", "bluestore_bdev_size": "107268255744", "bluestore_bdev_type": "hdd", "ceph_version": "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", "cpu": "Intel(R) Xeon(R) CPU E7-4820 v3 @ 1.90GHz", "default_device_class": "hdd", "distro": "ubuntu", "distro_description": "Ubuntu 18.04.3 LTS", "distro_version": "18.04", "front_addr": "172.18.2.178:6800/12605", "front_iface": "ens160", "hb_back_addr": "172.18.2.178:6802/12605", "hb_front_addr": "172.18.2.178:6803/12605", "hostname": "ceph", "journal_rotational": "1", "kernel_description": "#92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020", "kernel_version": "4.15.0-91-generic", "mem_swap_kb": "4194300", "mem_total_kb": "8168160", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "rotational": "1" } there are paths and size. does bdev mean block in ceph-bluestore-tool? ceph daemon osd.0 config show ( block "bluestore_block_path": "", "bluestore_block_size": "10737418240", db "bluestore_block_db_create": "false", "bluestore_block_db_path": "", "bluestore_block_db_size": "0", wal "bluestore_block_wal_create": "false", "bluestore_block_wal_path": "", "bluestore_block_wal_size": "100663296", there is no path info and only have wal size What is the best way to get the path and size infomation of block,block.db adn block.wal? linyunfan Igor Fedotov <ifedotov@suse.de> 于2020年5月5日周二 下午10:47写道：
Hi Dave,
wouldn't this help (particularly "Viewing runtime settings" section):
https://docs.ceph.com/docs/nautilus/rados/configuration/ceph-conf/
Thanks,
Igor
On 5/5/2020 2:52 AM, Dave Hall wrote:
Hello,
Sorry if this has been asked before...
A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance.
While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware.
I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration.
Could anybody point me to a command or link to documentation on this?
Thanks.
-Dave
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Lin, Igor, Herve, With the help of the information all of you have provided I have now reviewed the massive amount of information that is available for just one of my OSDs. (All 24 were configured with Ceph-Ansible, so all should be configured the same.) Like Lin's example below, I see that I don't have a WAL device. This is a place where the config documentation was a bit confusing: Each OSD node has 8 x 12TB SAS drives and a 1.6TB PCIe NVMe. Since there are 4 additional bays I set a parameter in Ceph-Ansible to divide the NVMe into 12 'slices'. When I was setting this up there was something in the docs that made it seem that by putting the DB on the NVMe, Bluestore would automatically use extra space in the NVMe slice for WAL. Is this correct, or does it have to be specified in some explicit way? Further, I'm trying to understand how my NVMe is 'sliced' or 'partitioned'. It doesn't seem to have a partition table so maybe it's just one big LVM device. Nevertheless, I'd like to be able to find out how the NVMe is sliced up, how big the Rocks DB is, and whether any extra space is being used for WAL or anything else. Of course, if I've failed to create an optimal configuration I'm next going to ask if I can adjust it without having to wipe and reinitialize every OSD. Thanks. -Dave Dave Hall Binghamton University kdhall@binghamton.edu On 5/6/2020 2:20 AM, lin yunfan wrote:
Is there a way to get the block,block.db,block.wal path and size? what if all of them or some of them are colocated in one disk?
I can get the info from a wal,db,block colocated osd like below:
ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0/ { "/var/lib/ceph/osd/ceph-0//block": { "osd_uuid": "199f2445-af9e-4172-8231-6d98858684e8", "size": 107268255744, "btime": "2020-02-15 21:53:42.972004", "description": "main", "bluefs": "1", "ceph_fsid": "83a73817-3566-4044-91b6-22cee6753515", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "ready": "ready", "require_osd_release": "\u000c", "whoami": "0" } } but there is no path of the block.
ceph osd metadata 0 { "id": 0, "arch": "x86_64", "back_addr": "172.18.2.178:6801/12605", "back_iface": "ens160", "bluefs": "1", "bluefs_db_access_mode": "blk", "bluefs_db_block_size": "4096", "bluefs_db_dev": "8:16", "bluefs_db_dev_node": "sdb", "bluefs_db_driver": "KernelDevice", "bluefs_db_model": "Virtual disk ", "bluefs_db_partition_path": "/dev/sdb2", "bluefs_db_rotational": "1", "bluefs_db_size": "107268255744", "bluefs_db_type": "hdd", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev": "8:16", "bluestore_bdev_dev_node": "sdb", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_model": "Virtual disk ", "bluestore_bdev_partition_path": "/dev/sdb2", "bluestore_bdev_rotational": "1", "bluestore_bdev_size": "107268255744", "bluestore_bdev_type": "hdd", "ceph_version": "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)", "cpu": "Intel(R) Xeon(R) CPU E7-4820 v3 @ 1.90GHz", "default_device_class": "hdd", "distro": "ubuntu", "distro_description": "Ubuntu 18.04.3 LTS", "distro_version": "18.04", "front_addr": "172.18.2.178:6800/12605", "front_iface": "ens160", "hb_back_addr": "172.18.2.178:6802/12605", "hb_front_addr": "172.18.2.178:6803/12605", "hostname": "ceph", "journal_rotational": "1", "kernel_description": "#92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020", "kernel_version": "4.15.0-91-generic", "mem_swap_kb": "4194300", "mem_total_kb": "8168160", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "rotational": "1" } there are paths and size. does bdev mean block in ceph-bluestore-tool?
ceph daemon osd.0 config show ( block "bluestore_block_path": "", "bluestore_block_size": "10737418240",
db "bluestore_block_db_create": "false", "bluestore_block_db_path": "", "bluestore_block_db_size": "0",
wal "bluestore_block_wal_create": "false", "bluestore_block_wal_path": "", "bluestore_block_wal_size": "100663296", there is no path info and only have wal size
What is the best way to get the path and size infomation of block,block.db adn block.wal?
linyunfan
Igor Fedotov <ifedotov@suse.de> 于2020年5月5日周二 下午10:47写道：
Hi Dave,
wouldn't this help (particularly "Viewing runtime settings" section):
https://docs.ceph.com/docs/nautilus/rados/configuration/ceph-conf/
Thanks,
Igor
On 5/5/2020 2:52 AM, Dave Hall wrote:
Hello,
Sorry if this has been asked before...
A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance.
While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware.
I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration.
Could anybody point me to a command or link to documentation on this?
Thanks.
-Dave
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Igor, Thank you for pointing this section out to me. The information there is useful, but I would comment for John Dover's interest that the section was hard to find. I'd been on that page several times before, but became overwhelmed by the time I got that far down. Maybe the 'What do I have' section should be at the top and the 'What can I do with it' section should come second? (My personal opinion, of course). It is certainly a challenge with a software project this vast to figure out how to organize the documentation so that it's easy for the unfamiliar to find what they need. -Dave Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On 5/5/2020 10:42 AM, Igor Fedotov wrote:
Hi Dave,
wouldn't this help (particularly "Viewing runtime settings" section):
https://docs.ceph.com/docs/nautilus/rados/configuration/ceph-conf/
Thanks,
Igor
On 5/5/2020 2:52 AM, Dave Hall wrote:
Hello,
Sorry if this has been asked before...
A few months ago I deployed a small Nautilus cluster using ceph-ansible. The OSD nodes have multiple spinning drives and a PCI NVMe. Now that the cluster has been stable for a while it's time to start optimizing performance.
While I can tell that there is a part of the NVMe associated with each OSD, I'm trying to verify which BlueStore components are using the NVMe - WAL, DB, Cache - and whether the configuration generated by ceph-ansible (and my settings in osds.yml) is optimal for my hardware.
I've searched around a bit and, while I have found documentation on how to configure, reconfigure, and repair a BlueStore OSD, I haven't found anything on how to query the current configuration.
Could anybody point me to a command or link to documentation on this?
Thanks.
-Dave
participants (5)
-
Dave Hall
-
Herve Ballans
-
Igor Fedotov
-
John Zachary Dover
-
lin yunfan