Move block.db to new ssd
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions. The new 4TB SSD's I have split into volumes with: # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 & osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this: # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami I then take osd4 out of the cluster and stop it. Then I do this: # ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami Thereafter I do this to load the changes: # ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 --path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami After that the configuration change has reverted to what it was before. I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 What am I missing to get these changes to be permanent? regards Roland
Hi Roland, On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD. The OSD's that need to have their journals moved are osd.5, osd.4, osd.3
& osd.2. I did osd.5 by deleting the osd and recreating it. However, that takes a lot of time with a spinner, so I'd rather just move the Rocksdb and WAL. Of course, this needs to be done on all the nodes. The osd.4 directory looks like this:
# ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block ->
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami
I then take osd4 out of the cluster and stop it. Then I do this:
# ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-4/ --dev-target /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source /var/lib/ceph/osd/ceph-4/block.db inferring bluefs devices from bluestore path device removed:1 /var/lib/ceph/osd/ceph-4/block.db device added: 1 /dev/dm-4 # ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block ->
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring -rw------- 1 ceph ceph 6 Nov 10 14:36 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 10 14:36 type -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami
Thereafter I do this to load the changes:
# ceph-volume lvm activate --all --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is active. Skipping activation --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is active. Skipping activation --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is active. Skipping activation --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is active. Skipping activation --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is active. Skipping activation --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is active. Skipping activation --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is active. Skipping activation --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
--path /var/lib/ceph/osd/ceph-4 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
/var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 Running command: /usr/bin/ln -snf /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 Running command: /usr/bin/systemctl enable ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 Running command: /usr/bin/systemctl start ceph-osd@4 --> ceph-volume lvm activate successful for osd ID: 4 --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is active. Skipping activation --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is active. Skipping activation --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is active. Skipping activation
# ls -la /var/lib/ceph/osd/ceph-4/ total 32 drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block ->
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring -rw------- 1 ceph ceph 6 Nov 11 21:09 ready -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release -rw------- 1 ceph ceph 10 Nov 11 21:09 type -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami
After that the configuration change has reverted to what it was before.
I have tried using the following, but it doesn't help: # lvchange --deltag "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2"
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 # lvchange --addtag "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4"
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/ Cheers, Alwin --
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/
On 2024/11/12 04:54, Alwin Antreich wrote:
Hi Roland,
On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD.
I see relatively small RocksDB and not WAL? ceph daemon osd.4 perf dump <snip> "bluefs": { "db_total_bytes": 45025845248, "db_used_bytes": 2131755008, "wal_total_bytes": 0, "wal_used_bytes": 0, </snip> I have been led to understand that 4% is die high end and only on very busy systems is that reached, if ever?
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
Ah, excellent! I didn't find that in my searches. Will try that now. regards Roland
Cheers, Alwin
--
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/12 04:54, Alwin Antreich wrote:
Hi Roland,
On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD.
I see relatively small RocksDB and not WAL?
ceph daemon osd.4 perf dump <snip> "bluefs": { "db_total_bytes": 45025845248, "db_used_bytes": 2131755008, "wal_total_bytes": 0, "wal_used_bytes": 0, </snip>
I have been led to understand that 4% is die high end and only on very busy systems is that reached, if ever?
Hi Roland, This is generally true but it depends on what your cluster is used for. If your cluster is used for block (RBD) storage then 1%-2% should be enough. If your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather stay on the safe size and respect the 4% recommendation as these workloads make heavy use of block.db to store metadata. Now percentage is one thing, level size is another. To avoid overspilling when block.db size approaches 30GB you'd better choose a block.db size of 300GB+ whatever the percentage of block size this is, if you don't want to play with rocksdb level size and multiplier, which you probably don't. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si... [2] https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider... [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
Ah, excellent! I didn't find that in my searches. Will try that now.
regards
Roland
Cheers, Alwin
--
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Frédéric, The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph 15.2.8, due to the new default (bluestore_volume_selection_policy = use_some_extra), it no longer wastes the extra capacity of the DB device. On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/12 04:54, Alwin Antreich wrote:
Hi Roland,
On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD.
I see relatively small RocksDB and not WAL?
ceph daemon osd.4 perf dump <snip> "bluefs": { "db_total_bytes": 45025845248, "db_used_bytes": 2131755008, "wal_total_bytes": 0, "wal_used_bytes": 0, </snip>
I have been led to understand that 4% is die high end and only on very busy systems is that reached, if ever?
Hi Roland,
This is generally true but it depends on what your cluster is used for.
If your cluster is used for block (RBD) storage then 1%-2% should be enough. If your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather stay on the safe size and respect the 4% recommendation as these workloads make heavy use of block.db to store metadata.
Now percentage is one thing, level size is another. To avoid overspilling when block.db size approaches 30GB you'd better choose a block.db size of 300GB+ whatever the percentage of block size this is, if you don't want to play with rocksdb level size and multiplier, which you probably don't.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si... [2] https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider... [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
Ah, excellent! I didn't find that in my searches. Will try that now.
regards
Roland
Cheers, Alwin
--
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Hello Alexander, Thank you for clarifying this point. The documentation was not very clear about the 'improvements'. Does that mean that in the latest releases overspilling no longer occurs between the two thresholds of 30GB and 300GB? Meaning block.db can be 80GB in size without overspilling, for example? Cheers, Frédéric. ----- Le 12 Nov 24, à 13:32, Alexander Patrakov patrakov@gmail.com a écrit :
Hello Frédéric,
The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph 15.2.8, due to the new default (bluestore_volume_selection_policy = use_some_extra), it no longer wastes the extra capacity of the DB device.
On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/12 04:54, Alwin Antreich wrote:
Hi Roland,
On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD.
I see relatively small RocksDB and not WAL?
ceph daemon osd.4 perf dump <snip> "bluefs": { "db_total_bytes": 45025845248, "db_used_bytes": 2131755008, "wal_total_bytes": 0, "wal_used_bytes": 0, </snip>
I have been led to understand that 4% is die high end and only on very busy systems is that reached, if ever?
Hi Roland,
This is generally true but it depends on what your cluster is used for.
If your cluster is used for block (RBD) storage then 1%-2% should be enough. If your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather stay on the safe size and respect the 4% recommendation as these workloads make heavy use of block.db to store metadata.
Now percentage is one thing, level size is another. To avoid overspilling when block.db size approaches 30GB you'd better choose a block.db size of 300GB+ whatever the percentage of block size this is, if you don't want to play with rocksdb level size and multiplier, which you probably don't.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si... [2] https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider... [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
Ah, excellent! I didn't find that in my searches. Will try that now.
regards
Roland
Cheers, Alwin
--
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
Yes, that is correct. On Tue, Nov 12, 2024 at 8:51 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hello Alexander,
Thank you for clarifying this point. The documentation was not very clear about the 'improvements'.
Does that mean that in the latest releases overspilling no longer occurs between the two thresholds of 30GB and 300GB? Meaning block.db can be 80GB in size without overspilling, for example?
Cheers, Frédéric.
----- Le 12 Nov 24, à 13:32, Alexander Patrakov patrakov@gmail.com a écrit :
Hello Frédéric,
The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph 15.2.8, due to the new default (bluestore_volume_selection_policy = use_some_extra), it no longer wastes the extra capacity of the DB device.
On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/12 04:54, Alwin Antreich wrote:
Hi Roland,
On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD.
I see relatively small RocksDB and not WAL?
ceph daemon osd.4 perf dump <snip> "bluefs": { "db_total_bytes": 45025845248, "db_used_bytes": 2131755008, "wal_total_bytes": 0, "wal_used_bytes": 0, </snip>
I have been led to understand that 4% is die high end and only on very busy systems is that reached, if ever?
Hi Roland,
This is generally true but it depends on what your cluster is used for.
If your cluster is used for block (RBD) storage then 1%-2% should be enough. If your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather stay on the safe size and respect the 4% recommendation as these workloads make heavy use of block.db to store metadata.
Now percentage is one thing, level size is another. To avoid overspilling when block.db size approaches 30GB you'd better choose a block.db size of 300GB+ whatever the percentage of block size this is, if you don't want to play with rocksdb level size and multiplier, which you probably don't.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si... [2] https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider... [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
Ah, excellent! I didn't find that in my searches. Will try that now.
regards
Roland
Cheers, Alwin
--
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov
Hi Anthony, Did the RocksDB sharding end up improving the overspilling situation related to the level thresholds? I had only anticipated that it would reduce the impact of compaction. We reshared our OSD's RocksDBs a long time ago (after upgrading to Pacific IIRC) and I think we could still observe overspilling at the layer levels sometimes, if I'm not mistaken. Cheers, Frédéric. PS: It seems that the document you referred to is not accessible from the Internet. ----- Le 12 Nov 24, à 15:11, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
RocksDB column sharding came a while ago. It should be enabled on your OSDs, provided they weren’t built on a much older release. If they were you can update them.
[ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... ] [ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... | rocksdb_in_ceph ] [ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... | PDF Document · 512 KB ]
[ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... | IBM Storage Ceph – Administration, Resharding RocksDB database reshard RocksDB database ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... | ibm.com ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... ]
On Nov 12, 2024, at 8:02 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Yes, that is correct.
On Tue, Nov 12, 2024 at 8:51 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hello Alexander,
Thank you for clarifying this point. The documentation was not very clear about the 'improvements'.
Does that mean that in the latest releases overspilling no longer occurs between the two thresholds of 30GB and 300GB? Meaning block.db can be 80GB in size without overspilling, for example?
Cheers,
Frédéric.
----- Le 12 Nov 24, à 13:32, Alexander Patrakov patrakov@gmail.com a écrit :
Hello Frédéric,
The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph
15.2.8, due to the new default (bluestore_volume_selection_policy =
use_some_extra), it no longer wastes the extra capacity of the DB
device.
On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass
<frederic.nass@univ-lorraine.fr> wrote:
----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/12 04:54, Alwin Antreich wrote:
> Hi Roland,
> On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
>> I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's
>> who are end of life. I have some spinners who have their journals on
>> SSD. Each spinner has a 50GB SSD LVM partition and I want to move those
>> each to new corresponding partitions.
>> The new 4TB SSD's I have split into volumes with:
>> # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0
>> # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0
>> # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0
>> # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0
>> # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1
>> # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
> I caution the mix of DB/WAL partitions with other applications. The
> performance profile may not be suited for shared use. And depending on the
> use case the ~48GB might not be big enough to hinder DB spillover. See the
> current size when querying the OSD.
I see relatively small RocksDB and not WAL?
ceph daemon osd.4 perf dump
<snip>
"bluefs": {
"db_total_bytes": 45025845248,
"db_used_bytes": 2131755008,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
</snip>
I have been led to understand that 4% is die high end and only on very busy
systems is that reached, if ever?
Hi Roland,
This is generally true but it depends on what your cluster is used for.
If your cluster is used for block (RBD) storage then 1%-2% should be enough. If
your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather
stay on the safe size and respect the 4% recommendation as these workloads make
heavy use of block.db to store metadata.
Now percentage is one thing, level size is another. To avoid overspilling when
block.db size approaches 30GB you'd better choose a block.db size of 300GB+
whatever the percentage of block size this is, if you don't want to play with
rocksdb level size and multiplier, which you probably don't.
Regards,
Frédéric.
[1]
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si...
[2]
https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider...
[3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
>> What am I missing to get these changes to be permanent?
> Likely just an issue with the order of execution. But there is an easier
> way to do the move. See:
Ah, excellent! I didn't find that in my searches. Will try that now.
regards
Roland
> Cheers,
> Alwin
> --
>> Alwin Antreich
> Head of Training and Proxmox Services
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io/
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
--
Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, it improves the dynamic where only ~3, 30, 300, etc. GB of DB space can be used, and thus mitigates spillover. Previously a, say, 29GB DB device/partition would be like 85% unused. With recent releases one can also turn on DB compression, which should have a similar benefit.
On Nov 12, 2024, at 11:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Anthony,
Did the RocksDB sharding end up improving the overspilling situation related to the level thresholds? I had only anticipated that it would reduce the impact of compaction.
We reshared our OSD's RocksDBs a long time ago (after upgrading to Pacific IIRC) and I think we could still observe overspilling at the layer levels sometimes, if I'm not mistaken.
Cheers, Frédéric.
PS: It seems that the document you referred to is not accessible from the Internet.
----- Le 12 Nov 24, à 15:11, Anthony D'Atri <anthony.datri@gmail.com> a écrit : RocksDB column sharding came a while ago. It should be enabled on your OSDs, provided they weren’t built on a much older release. If they were you can update them.

rocksdb_in_ceph PDF Document · 512 KB <https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2cd/ceph/rocksdb_in_ceph.pdf>
IBM Storage Ceph – Administration, Resharding RocksDB database reshard RocksDB database ibm.com <https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rocksdb-database> <https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rocksdb-database>
On Nov 12, 2024, at 8:02 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Yes, that is correct.
On Tue, Nov 12, 2024 at 8:51 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hello Alexander,
Thank you for clarifying this point. The documentation was not very clear about the 'improvements'.
Does that mean that in the latest releases overspilling no longer occurs between the two thresholds of 30GB and 300GB? Meaning block.db can be 80GB in size without overspilling, for example?
Cheers, Frédéric.
----- Le 12 Nov 24, à 13:32, Alexander Patrakov patrakov@gmail.com a écrit :
Hello Frédéric,
The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph 15.2.8, due to the new default (bluestore_volume_selection_policy = use_some_extra), it no longer wastes the extra capacity of the DB device.
On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/12 04:54, Alwin Antreich wrote: Hi Roland,
On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's who are end of life. I have some spinners who have their journals on SSD. Each spinner has a 50GB SSD LVM partition and I want to move those each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD.
I see relatively small RocksDB and not WAL?
ceph daemon osd.4 perf dump <snip> "bluefs": { "db_total_bytes": 45025845248, "db_used_bytes": 2131755008, "wal_total_bytes": 0, "wal_used_bytes": 0, </snip>
I have been led to understand that 4% is die high end and only on very busy systems is that reached, if ever?
Hi Roland,
This is generally true but it depends on what your cluster is used for.
If your cluster is used for block (RBD) storage then 1%-2% should be enough. If your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather stay on the safe size and respect the 4% recommendation as these workloads make heavy use of block.db to store metadata.
Now percentage is one thing, level size is another. To avoid overspilling when block.db size approaches 30GB you'd better choose a block.db size of 300GB+ whatever the percentage of block size this is, if you don't want to play with rocksdb level size and multiplier, which you probably don't.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si... [2] https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider... [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
What am I missing to get these changes to be permanent?
Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
Ah, excellent! I didn't find that in my searches. Will try that now.
regards
Roland
Cheers, Alwin
--
Alwin Antreich Head of Training and Proxmox Services
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yep, we're using RocksDB compression with Pacific since a few month. It helped a lot. Since we're talking overspilling... Despite using bluestore_volume_selection_policy=use_some_extra with resharded RocksDB databases we can still observe many OSDs overspilling from time to time (approximately every month and a half). When this happens: - almost all OSDs overspill one after the other over 2-3 days. They all get detected and compacted thanks to a cron job, then it's completely quiet again for another month and a half, and then it comes back. This phenomenon repeats cyclically. - 'ceph health detail' shows figures similar to thoses reported in [1] that [2] is supposed to have fixed (if I'm not mistaken): === Full health status === [WARN] BLUEFS_SPILLOVER: 8 OSD(s) experiencing BlueFS spillover osd.337 spilled over 12 GiB metadata from 'db' device (12 GiB used of 124 GiB) to slow device osd.352 spilled over 12 GiB metadata from 'db' device (687 MiB used of 124 GiB) to slow device osd.353 spilled over 12 GiB metadata from 'db' device (152 MiB used of 124 GiB) to slow device osd.357 spilled over 12 GiB metadata from 'db' device (960 MiB used of 124 GiB) to slow device osd.359 spilled over 1.9 GiB metadata from 'db' device (12 GiB used of 124 GiB) to slow device Has anyone ever experienced this? Cheers, Frédéric. [1] [ https://tracker.ceph.com/issues/38745 | https://tracker.ceph.com/issues/38745 ] [2] [ https://github.com/ceph/ceph/pull/29687 | https://github.com/ceph/ceph/pull/29687 ] ----- Le 12 Nov 24, à 17:36, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
Yes, it improves the dynamic where only ~3, 30, 300, etc. GB of DB space can be used, and thus mitigates spillover. Previously a, say, 29GB DB device/partition would be like 85% unused. With recent releases one can also turn on DB compression, which should have a similar benefit.
On Nov 12, 2024, at 11:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Anthony,
Did the RocksDB sharding end up improving the overspilling situation related to the level thresholds? I had only anticipated that it would reduce the impact of compaction.
We reshared our OSD's RocksDBs a long time ago (after upgrading to Pacific IIRC) and I think we could still observe overspilling at the layer levels sometimes, if I'm not mistaken.
Cheers, Frédéric.
PS: It seems that the document you referred to is not accessible from the Internet.
----- Le 12 Nov 24, à 15:11, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
RocksDB column sharding came a while ago. It should be enabled on your OSDs, provided they weren’t built on a much older release. If they were you can update them.
[ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... ] [ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... | rocksdb_in_ceph ] [ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... | PDF Document · 512 KB ]
[ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... | IBM Storage Ceph – Administration, Resharding RocksDB database reshard RocksDB database ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... | ibm.com ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... ]
On Nov 12, 2024, at 8:02 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Yes, that is correct.
On Tue, Nov 12, 2024 at 8:51 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hello Alexander,
Thank you for clarifying this point. The documentation was not very clear about the 'improvements'.
Does that mean that in the latest releases overspilling no longer occurs between the two thresholds of 30GB and 300GB? Meaning block.db can be 80GB in size without overspilling, for example?
Cheers,
Frédéric.
----- Le 12 Nov 24, à 13:32, Alexander Patrakov patrakov@gmail.com a écrit :
Hello Frédéric,
The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph
15.2.8, due to the new default (bluestore_volume_selection_policy =
use_some_extra), it no longer wastes the extra capacity of the DB
device.
On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass
<frederic.nass@univ-lorraine.fr> wrote:
> ----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
>> On 2024/11/12 04:54, Alwin Antreich wrote:
>>> Hi Roland,
>>> On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
>>>> I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's
>>>> who are end of life. I have some spinners who have their journals on
>>>> SSD. Each spinner has a 50GB SSD LVM partition and I want to move those
>>>> each to new corresponding partitions.
>>>> The new 4TB SSD's I have split into volumes with:
>>>> # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0
>>>> # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0
>>>> # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0
>>>> # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0
>>>> # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1
>>>> # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
>>> I caution the mix of DB/WAL partitions with other applications. The
>>> performance profile may not be suited for shared use. And depending on the
>>> use case the ~48GB might not be big enough to hinder DB spillover. See the
>>> current size when querying the OSD.
>> I see relatively small RocksDB and not WAL?
>> ceph daemon osd.4 perf dump
>> <snip>
>> "bluefs": {
>> "db_total_bytes": 45025845248,
>> "db_used_bytes": 2131755008,
>> "wal_total_bytes": 0,
>> "wal_used_bytes": 0,
>> </snip>
>> I have been led to understand that 4% is die high end and only on very busy
>> systems is that reached, if ever?
> Hi Roland,
> This is generally true but it depends on what your cluster is used for.
> If your cluster is used for block (RBD) storage then 1%-2% should be enough. If
> your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather
> stay on the safe size and respect the 4% recommendation as these workloads make
> heavy use of block.db to store metadata.
> Now percentage is one thing, level size is another. To avoid overspilling when
> block.db size approaches 30GB you'd better choose a block.db size of 300GB+
> whatever the percentage of block size this is, if you don't want to play with
> rocksdb level size and multiplier, which you probably don't.
> Regards,
> Frédéric.
> [1]
> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si...
> [2]
> https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider...
> [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
>>>> What am I missing to get these changes to be permanent?
>>> Likely just an issue with the order of execution. But there is an easier
>>> way to do the move. See:
>>> https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
>> Ah, excellent! I didn't find that in my searches. Will try that now.
>> regards
>> Roland
>>> Cheers,
>>> Alwin
>>> --
>>>> Alwin Antreich
>>> Head of Training and Proxmox Services
>>> croit GmbH, Freseniusstr. 31h, 81247 Munich
>>> CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492
>>> Com. register: Amtsgericht Munich HRB 231263
>>> Web: https://croit.io/
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
--
Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, For the record, we have identified the root cause of the overspilling issue. Previously, an ambiguity in the RocksDB resharding documentation led us to reshard our RocksDB databases using a lowercase 'o' instead of an uppercase 'O' in the command: $ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} L P" reshard instead of $ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard As a consequence, the 'Object' key/value pairs were stored in the default column family instead of the intended 'O' column family. Resharding again with the correct capital 'O' resolved this issue for the affected overspilling OSDs. Be aware that there remains an error in both the Quincy and Reef documentation, where the letters 'l' and 'p' should actually be uppercase 'L' and 'P'. Using lowercase 'l' and 'p' wrecks the OSD entirely. I've created this tracker [1] in hopes of getting the Quincy and Reef documentation fixed. Regards, Frédéric. [1] https://tracker.ceph.com/issues/70185 ----- Le 12 Nov 24, à 22:56, Frédéric Nass frederic.nass@univ-lorraine.fr a écrit :
Yep, we're using RocksDB compression with Pacific since a few month. It helped a lot.
Since we're talking overspilling... Despite using bluestore_volume_selection_policy=use_some_extra with resharded RocksDB databases we can still observe many OSDs overspilling from time to time (approximately every month and a half). When this happens:
- almost all OSDs overspill one after the other over 2-3 days. They all get detected and compacted thanks to a cron job, then it's completely quiet again for another month and a half, and then it comes back. This phenomenon repeats cyclically. - 'ceph health detail' shows figures similar to thoses reported in [1] that [2] is supposed to have fixed (if I'm not mistaken):
=== Full health status === [WARN] BLUEFS_SPILLOVER: 8 OSD(s) experiencing BlueFS spillover osd.337 spilled over 12 GiB metadata from 'db' device (12 GiB used of 124 GiB) to slow device osd.352 spilled over 12 GiB metadata from 'db' device (687 MiB used of 124 GiB) to slow device osd.353 spilled over 12 GiB metadata from 'db' device (152 MiB used of 124 GiB) to slow device osd.357 spilled over 12 GiB metadata from 'db' device (960 MiB used of 124 GiB) to slow device osd.359 spilled over 1.9 GiB metadata from 'db' device (12 GiB used of 124 GiB) to slow device
Has anyone ever experienced this?
Cheers, Frédéric.
[1] [ https://tracker.ceph.com/issues/38745 | https://tracker.ceph.com/issues/38745 ] [2] [ https://github.com/ceph/ceph/pull/29687 | https://github.com/ceph/ceph/pull/29687 ]
----- Le 12 Nov 24, à 17:36, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
Yes, it improves the dynamic where only ~3, 30, 300, etc. GB of DB space can be used, and thus mitigates spillover. Previously a, say, 29GB DB device/partition would be like 85% unused. With recent releases one can also turn on DB compression, which should have a similar benefit.
On Nov 12, 2024, at 11:25 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi Anthony,
Did the RocksDB sharding end up improving the overspilling situation related to the level thresholds? I had only anticipated that it would reduce the impact of compaction.
We reshared our OSD's RocksDBs a long time ago (after upgrading to Pacific IIRC) and I think we could still observe overspilling at the layer levels sometimes, if I'm not mistaken.
Cheers, Frédéric.
PS: It seems that the document you referred to is not accessible from the Internet.
----- Le 12 Nov 24, à 15:11, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
RocksDB column sharding came a while ago. It should be enabled on your OSDs, provided they weren’t built on a much older release. If they were you can update them.
[ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... ] [ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... | rocksdb_in_ceph ] [ https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2... | PDF Document · 512 KB ]
[ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... | IBM Storage Ceph – Administration, Resharding RocksDB database reshard RocksDB database ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... | ibm.com ] [ https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-resharding-rock... ]
On Nov 12, 2024, at 8:02 AM, Alexander Patrakov <patrakov@gmail.com> wrote:
Yes, that is correct.
On Tue, Nov 12, 2024 at 8:51 PM Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hello Alexander,
Thank you for clarifying this point. The documentation was not very clear about the 'improvements'.
Does that mean that in the latest releases overspilling no longer occurs between the two thresholds of 30GB and 300GB? Meaning block.db can be 80GB in size without overspilling, for example?
Cheers,
Frédéric.
----- Le 12 Nov 24, à 13:32, Alexander Patrakov patrakov@gmail.com a écrit :
> Hello Frédéric,
> The advice regarding 30/300 GB DB sizes is no longer valid. Since Ceph
> 15.2.8, due to the new default (bluestore_volume_selection_policy =
> use_some_extra), it no longer wastes the extra capacity of the DB
> device.
> On Tue, Nov 12, 2024 at 5:52 PM Frédéric Nass
> <frederic.nass@univ-lorraine.fr> wrote:
>> ----- Le 12 Nov 24, à 8:51, Roland Giesler roland@giesler.za.net a écrit :
>>> On 2024/11/12 04:54, Alwin Antreich wrote:
>>>> Hi Roland,
>>>> On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@giesler.za.net> wrote:
>>>>> I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's
>>>>> who are end of life. I have some spinners who have their journals on
>>>>> SSD. Each spinner has a 50GB SSD LVM partition and I want to move those
>>>>> each to new corresponding partitions.
>>>>> The new 4TB SSD's I have split into volumes with:
>>>>> # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0
>>>>> # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0
>>>>> # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0
>>>>> # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0
>>>>> # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1
>>>>> # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
>>>> I caution the mix of DB/WAL partitions with other applications. The
>>>> performance profile may not be suited for shared use. And depending on the
>>>> use case the ~48GB might not be big enough to hinder DB spillover. See the
>>>> current size when querying the OSD.
>>> I see relatively small RocksDB and not WAL?
>>> ceph daemon osd.4 perf dump
>>> <snip>
>>> "bluefs": {
>>> "db_total_bytes": 45025845248,
>>> "db_used_bytes": 2131755008,
>>> "wal_total_bytes": 0,
>>> "wal_used_bytes": 0,
>>> </snip>
>>> I have been led to understand that 4% is die high end and only on very busy
>>> systems is that reached, if ever?
>> Hi Roland,
>> This is generally true but it depends on what your cluster is used for.
>> If your cluster is used for block (RBD) storage then 1%-2% should be enough. If
>> your cluster is used for file (cephfs) and S3 (RGW) storage then you'd rather
>> stay on the safe size and respect the 4% recommendation as these workloads make
>> heavy use of block.db to store metadata.
>> Now percentage is one thing, level size is another. To avoid overspilling when
>> block.db size approaches 30GB you'd better choose a block.db size of 300GB+
>> whatever the percentage of block size this is, if you don't want to play with
>> rocksdb level size and multiplier, which you probably don't.
>> Regards,
>> Frédéric.
>> [1]
>> https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#si...
>> [2]
>> https://www.ibm.com/docs/en/storage-ceph/7.1?topic=bluestore-sizing-consider...
>> [3] https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
>>>>> What am I missing to get these changes to be permanent?
>>>> Likely just an issue with the order of execution. But there is an easier
>>>> way to do the move. See:
>>>> https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/
>>> Ah, excellent! I didn't find that in my searches. Will try that now.
>>> regards
>>> Roland
>>>> Cheers,
>>>> Alwin
>>>> --
>>>>> Alwin Antreich
>>>> Head of Training and Proxmox Services
>>>> croit GmbH, Freseniusstr. 31h, 81247 Munich
>>>> CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492
>>>> Com. register: Amtsgericht Munich HRB 231263
>>>> Web: https://croit.io/
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users@ceph.io
>>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-leave@ceph.io
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-leave@ceph.io
> --
> Alexander Patrakov
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Am 3/11/25 um 11:33 schrieb Frédéric Nass:
$ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard
As a consequence, the 'Object' key/value pairs were stored in the default column family instead of the intended 'O' column family. Resharding again with the correct capital 'O' resolved this issue for the affected overspilling OSDs.
Be aware that there remains an error in both the Quincy and Reef documentation, where the letters 'l' and 'p' should actually be uppercase 'L' and 'P'. Using lowercase 'l' and 'p' wrecks the OSD entirely. I've created this tracker [1] in hopes of getting the Quincy and Reef documentation fixed.
The latest docs seems to have it correct: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro... Is there a reason why you omitted "=min_write_buffer_number_to_merge=32" after L and P? These are the default values that I also see on my reef cluster. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi Robert, Thanks for pointing that out. This issue stems from version differences between our cluster environments: Our current Pacific cluster (migrating to Reef next week) uses the default configuration: bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P' While Reef and newer versions use: bluestore_rocksdb_cfs = 'm(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32' According to documentation, the bluestore_rocksdb_cfs parameter is only applied during OSD creation (when using --mkfs). Since there was no mention of needing to include L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 in the reshard command, I didn't include these options when resharding the OSDs. Did you? Additionally, min_write_buffer_number_to_merge appears to be a RocksDB option (bluestore_rocksdb_options) with a default value of 1 in Pacific and 6 in Reef. It's unclear whether the effective value is determined by bluestore_rocksdb_options or by the value specified in bluestore_rocksdb_cfs during OSD creation. This probably deserves clarification to prevent inconsistent behavior within the same cluster due to OSDs having different min_write_buffer_number_to_merge values from their creation time. Hopefully Igor can provide some insight on this issue. Regards, Frédéric ----- Le 11 Mar 25, à 12:05, Robert Sander r.sander@heinlein-support.de a écrit :
Hi,
Am 3/11/25 um 11:33 schrieb Frédéric Nass:
$ cephadm shell --name osd.OSD_ID --fsid $(ceph fsid) ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-OSD_ID --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard
As a consequence, the 'Object' key/value pairs were stored in the default column family instead of the intended 'O' column family. Resharding again with the correct capital 'O' resolved this issue for the affected overspilling OSDs.
Be aware that there remains an error in both the Quincy and Reef documentation, where the letters 'l' and 'p' should actually be uppercase 'L' and 'P'. Using lowercase 'l' and 'p' wrecks the OSD entirely. I've created this tracker [1] in hopes of getting the Quincy and Reef documentation fixed.
The latest docs seems to have it correct:
https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#ro...
Is there a reason why you omitted "=min_write_buffer_number_to_merge=32" after L and P? These are the default values that I also see on my reef cluster.
Regards -- Robert Sander Linux Consultant
Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Alexander Patrakov
-
Alwin Antreich
-
Anthony D'Atri
-
Frédéric Nass
-
Robert Sander
-
Roland Giesler