- ceph-users - lists.ceph.io

orphan multipart objects in Ceph cluster
by Ramin Najjarbashi 29 Mar '23

29 Mar '23

I hope this email finds you well. I wanted to share a recent experience I had with our Ceph cluster and get your feedback on a solution I came up with. Recently, we had some orphan objects stuck in our cluster that were not visible by any client like s3cmd, boto3, and mc. This caused some confusion for our users, as the sum of all objects in their buckets was much less than what we showed in the panel. We made some adjustments for them, but the issue persisted. As we have billions of objects in our cluster, using normal tools to find orphans was impossible. So, I came up with a tricky way to handle the situation. I created a bash script that identifies and removes the orphan objects using radosgw-admin and rados commands. Here is the script: https://gist.github.com/RaminNietzsche/b9baa06b69fc5f56d907f3c953769182 I am hoping to get some feedback from the community on this solution. Have any of you faced similar challenges with orphan objects in your Ceph clusters? Do you have any suggestions or improvements for my script? Thank you for your time and help.

2 1

osd_mclock_max_capacity_iops_ssd && multiple osd by nvme ?
by DERUMIER, Alexandre 29 Mar '23

29 Mar '23

Hi, I would like to advise to correctly tune osd_mclock_max_capacity_iops_ssd when you have multiple osd by nvme ? Does it need simply to divide the total iops of the nvme by number of osd ? But maybe it'll impact performance if more read/write are done on one of the osd ? I really don't known if we still need multiple osd by nvme in quincy ? Maybe it could be easier with a simple osd ? Regards, Alexandre

1 0

RGW can't create bucket
by kamil.madac＠gmail.com 29 Mar '23

29 Mar '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source

1 0

Adding new server to existing ceph cluster - with separate block.db on NVME
by Robert W. Eckert 29 Mar '23

29 Mar '23

Hi, I am trying to add a new server to an existing cluster, but cannot get the OSDs to create correctly When I try Cephadm ceph-volume lvm create, it returns nothing but the container info. [root@hiho ~]# cephadm ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Inferring fsid fe3a7cb0-69ca-11eb-8d45-c86000d08867 Using ceph image with id 'cc65afd6173a' and tag '<none>' created on 2022-10-17 23:41:41 +0000 UTC quay.io/ceph/ceph@sha256:2b73ccc9816e0a1ee1dfbe21ba9a8cc085210f1220f597b5050ebfcac4bdd346<mailto:quay.io/ceph/ceph@sha256:2b73ccc9816e0a1ee1dfbe21ba9a8cc085210f1220f597b5050ebfcac4bdd346> so I tried cephadm shell, and ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 41dafd4d-0579-4119-acca-6db31586a10f stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d0060d70) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d0063da0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-03-28T03:32:27.437+0000 7fa5d6253700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-03-28T03:32:27.437+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d6251ea0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-03-28T03:32:27.451+0000 7fa5ceffd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-03-28T03:32:27.453+0000 7fa5cf7fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-03-28T03:32:27.473+0000 7fa5cffff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-03-28T03:32:27.474+0000 7fa5d6253700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id I then copy the key ring file into the container using scp, but by that time the orchestrator created OSDs on the drives, so I have to delete the OSDs and start over. Then if I get the timing just right, I get this (from within cephadm shell): [ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e6e316d4-670d-4a9b-a50c-bc14d57394a3 Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-4d95584a-df28-4e21-9480-09a13f1fb804 /dev/sdd stdout: Physical volume "/dev/sdd" successfully created. stdout: Volume group "ceph-4d95584a-df28-4e21-9480-09a13f1fb804" successfully created Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 953861 -n osd-block-e6e316d4-670d-4a9b-a50c-bc14d57394a3 ceph-4d95584a-df28-4e21-9480-09a13f1fb804 stdout: Logical volume "osd-block-e6e316d4-670d-4a9b-a50c-bc14d57394a3" created. Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 119209 -n osd-db-9fc4f199-2c95-4ca7-a35c-ef4b08c86804 ceph-948a633c-420e-4f55-8515-b33e1c0ef18c stderr: Volume group "ceph-948a633c-420e-4f55-8515-b33e1c0ef18c" has insufficient free space (0 extents): 119209 required. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.12 --yes-i-really-mean-it stderr: purged osd.12 --> RuntimeError: Unable to find any LV for zapping OSD: 12 [ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 --> RuntimeError: Device /dev/sdd has a filesystem. [ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b93e1a8a-af88-431c-b705-f49d717b050f Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 /dev/sdd stdout: Physical volume "/dev/sdd" successfully created. stdout: Volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" successfully created Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 953861 -n osd-block-b93e1a8a-af88-431c-b705-f49d717b050f ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 stdout: Logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f" created. Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 /dev/nvme0n1p3 stdout: Physical volume "/dev/nvme0n1p3" successfully created. stdout: Volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" successfully created Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 119209 -n osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 stdout: Wiping ceph_bluestore signature on /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13. stdout: Logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Running command: /usr/bin/ln -s /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f /var/lib/ceph/osd/ceph-12/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-12/activate.monmap stderr: got monmap epoch 33 --> Creating keyring file for osd.12 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12/ Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8 Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 12 --monmap /var/lib/ceph/osd/ceph-12/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 --osd-data /var/lib/ceph/osd/ceph-12/ --osd-uuid b93e1a8a-af88-431c-b705-f49d717b050f --setuser ceph --setgroup ceph stderr: 2023-03-28T03:23:21.180+0000 7f0a40fd63c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/sdd Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f --path /var/lib/ceph/osd/ceph-12 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f /var/lib/ceph/osd/ceph-12/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/ln -snf /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 /var/lib/ceph/osd/ceph-12/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8 Running command: /usr/bin/systemctl enable ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f.service<mailto:/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f.service> -> /usr/lib/systemd/system/ceph-volume@.service<mailto:/usr/lib/systemd/system/ceph-volume@.service>. Running command: /usr/bin/systemctl enable --runtime ceph-osd@12 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)12.service<mailto:/run/systemd/system/ceph-osd.target.wants/ceph-osd@12.service> -> /usr/lib/systemd/system/ceph-osd@.service<mailto:/usr/lib/systemd/system/ceph-osd@.service>. Running command: /usr/bin/systemctl start ceph-osd@12 stderr: Failed to connect to bus: No such file or directory --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.12 --yes-i-really-mean-it stderr: purged osd.12 --> Zapping: /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f --> Unmounting /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-12 stderr: umount: /var/lib/ceph/osd/ceph-12 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0771422 s, 136 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 stderr: Removing ceph--e91c86b4--613f--45b0--b9d1--3bb76ed10f83-osd--block--b93e1a8a--af88--431c--b705--f49d717b050f (253:5) stderr: Releasing logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f" Archiving volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" metadata (seqno 5). stdout: Logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f" successfully removed. stderr: Removing physical volume "/dev/sdd" from volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" stdout: Volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" successfully removed stderr: Creating volume group backup "/etc/lvm/backup/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdd stdout: Labels on physical volume "/dev/sdd" successfully wiped. --> Zapping: /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131332 s, 798 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 stderr: Removing ceph--4abdb6f8--c891--43f8--8135--dd8470f80130-osd--db--5bab4d99--e2e5--48e7--aee6--d1a2103b9d13 (253:8) stderr: Releasing logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" Archiving volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" metadata (seqno 5). stdout: Logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" successfully removed. stderr: Removing physical volume "/dev/nvme0n1p3" from volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" stdout: Volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" successfully removed stderr: Creating volume group backup "/etc/lvm/backup/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme0n1p3 stdout: Labels on physical volume "/dev/nvme0n1p3" successfully wiped. --> Zapping successful for OSD: 12 --> RuntimeError: command returned non-zero exit status: 1 I have 3 other servers with the split data and block db that I was able to install just using cephadm, so I am not sure what is off on this one. Is there any way to either add the block.db to the already created OSDs or get around the missing bootstrap.osd/ceph or manually configure the disk and block db? I have also tried ceph orch apply osd --all-available-devices --unmanaged=true to stop ceph from trying to take the OSD, but it still does Thanks, Rob

2 3

s3-select introduction blog / Trino integration
by Gal Salomon 28 Mar '23

28 Mar '23

Hi https://ceph.io/en/news/blog/2022/s3select-intro/ Recently I published a blog on s3-select. the Blog discusses what it is, and why it is required. the last paragraph discusses the Trino(analytic SQL utility) and its integration with Ceph/s3select. that integration is still on-work, and it is quite promising. Trino does not just provides a comprehensive SQL but also provides scalable processing for the SQL statements. we will be glad to hear your ideas and comments. Gal.

1 0

Re: Ceph cluster out of balance after adding OSDs
by Robert Sander 28 Mar '23

28 Mar '23

On 27.03.23 16:34, Pat Vaughan wrote: > Yes, all the OSDs are using the SSD device class. Do you have multiple CRUSH rules by chance? Are all pools using the same CRUSH rule? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 3

Unexpected slow read for HDD cluster (good write speed)
by Rafael Weingartner 28 Mar '23

28 Mar '23

Hello guys! I would like to ask if somebody has already experienced a similar situation. We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. We are starting with a single dual port NIC (the bond has 50GiB in total), the design has been prepared so a new NIC can be added, and a new BOND can be created, where we intend to offload the cluster network. Therefore, logically speaking, we already configured different VLANs and networks for public and cluster traffic of Ceph. We are using Ubuntu 20.04 with Ceph Octopus. It is a standard deployment that we are used to. During our initial validations and evaluations of the cluster, we are reaching write speeds between 250-300MB/s, which would be the ballpark for this kind of setup for HDDs with the NVME as Rocks.db cache (in our experience). However, the issue is the reading process. While reading, we barely hit the mark of 100MB/s; we would expect at least something similar to the write speed. These tests are being performed in a pool with a replication factor of 3. We have already checked the disks, and they all seem to be reading just fine. The network does not seem to be the bottleneck either (checked with atop while reading/writing to the cluster). Have you guys ever encountered similar situations? Do you have any tips for us to proceed with the troubleshooting? We suspect that we are missing some small tuning detail, which is affecting the read performance only, but so far we could not pinpoint it. Any help would be much appreciated :)

4 4

Almalinux 9
by Sere Gerrit 28 Mar '23

28 Mar '23

Hello, Has anyone used almalinux 9 to install ceph. Have you encountered problems? Other tips on this installation are also welcome. Regards, Gerrit

4 3

Question about adding SSDs
by Kyriazis, George 27 Mar '23

27 Mar '23

Hello ceph community, We have a ceph cluster (Proxmox based) with is HDD-based. We’ve had some performance and “slow MDS” issues while doing VM/CT backups from the Proxmox cluster, especially when rebalancing is going on at the same time. My thought is that one of following is going to improve performance / response: 1. Add an M.2 drive for DB store on each node 2. Migrate the cephfs metadata pool to SSDs We have ~25 nodes with ~3 OSDs per node. (1) is a lot of work and will cost more. (2) seems more risky (to me) since the metadata pool would have to be migrated (potential loss in transit?) Which one of the 2 solutions above will give us more bang for the buck, or just plain better performance? I would hate to implement (1) to find out that another solution would’ve been better. Any other solutions that I haven’t thought of? Thank you! George

2 2

rbd cp vs. rbd clone + rbd flatten
by Tony Liu 27 Mar '23

27 Mar '23

Hi, I want 1) copy a snapshot to an image, 2) no need to copy snapshots, 3) no dependency after copy, 4) all same image format 2. In that case, is rbd cp the same as rbd clone + rbd flatten? I ran some tests, seems like it, but want to confirm, in case of missing anything. Also, seems cp is a bit faster and flatten, is that true? Thanks! Tony

2 2