- ceph-users - lists.ceph.io

Replace block drives of combined NVME+HDD OSDs
by Zakhar Kirpichenko 02 Apr '24

02 Apr '24

Hi, Unfortunately, some of our HDDs failed and we need to replace these drives which are parts of "combined" OSDs (DB/WAL on NVME, block storage on HDD). All OSDs are defined with a service definition similar to this one: ``` service_type: osd service_id: ceph02_combined_osd service_name: osd.ceph02_combined_osd placement: hosts: - ceph02 spec: data_devices: paths: - /dev/sda - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde - /dev/sdf - /dev/sdg - /dev/sdh - /dev/sdi db_devices: paths: - /dev/nvme0n1 - /dev/nvme1n1 filter_logic: AND objectstore: bluestore ``` In the above example, HDDs `sda` and `sdb` are not readable and data cannot be copied over to new HDDs. NVME partitions of `nvme0n1` with DB/WAL data are intact, but I guess that data is useless. I think the best approach is to replace the dead drives and completely rebuild each affected OSD. How should we go about this, preferably in a way that other OSDs on the node remain unaffected and operational? I would appreciate any advice or pointers to the relevant documentation. Best regards, Zakhar

2 3

Questions about rbd flatten command
by Henry lol 02 Apr '24

02 Apr '24

Hello, I executed multiple 'rbd flatten' commands simultaneously on a client. The elapsed time of each flatten job increased as the number of jobs increased, and network I/O was nearly full. so, I have two questions. 1. isn’t the flatten job running within the ceph cluster? Why is client-side network I/O so high? 2. How can I apply qos for each flatten job to reduce network I/O? Sincerely,

2 3

CEPH Quincy installation with multipathd enabled
by youssef.khristo＠elsewedyintel.com 02 Apr '24

02 Apr '24

Greetings community, we have a setup comprising of 6 servers hosting CentOS 8 Minimal Installation with CEPH Quincy version 18.2.2 supported by 20Gbps fiber optics NICs and a dual Xeon Intel processors, bootstrapped the installation on the first node then expanded to the others using the cephadm method, having the monitor services deployed on 5 of these nodes as well as 3 manager nodes. Each server has an NVMe boot disk as well as a 1TBs SATA SSD over which the OSDs are deployed. An EC profile was created with k=3 and m=3, serving a CephFS filesystem on top with NFS exports to serve other servers. Up to this point, the setup is quite stable in the sense that upon emergency reboot or network connection failure the OSDs did not fail and remain functional/started normally after reboot. At a certain point in our project, we had the need to activate the multipathd service, adding the boot drive partition and the CEPH SSD to its blacklist as to not be initialized for use by an mpath partition, the blacklist goes like so: boot blacklist: =============== blacklist { wwid "eui.<drive_id>" } SATA SSD blacklist: =================== blacklist { wwid "naa.<drive_id>" } The above blacklist configuration ensures that both the boot disk as well as CEPH's OSD function properly, with the following being lsblk output: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 894.3G 0 disk └─ceph--<id>-osd--block--<block_id> 252:3 0 894.3G 0 lvm nvme0n1 259:0 0 238.5G 0 disk ├─nvme0n1p1 259:1 0 600M 0 part /boot/efi ├─nvme0n1p2 259:2 0 1G 0 part /boot └─nvme0n1p3 259:3 0 236.9G 0 part ├─centos-root 252:0 0 170G 0 lvm / ├─centos-swap 252:1 0 23.4G 0 lvm [SWAP] ├─centos-var_log_audit 252:2 0 7.5G 0 lvm /var/log/audit ├─centos-home 252:4 0 26G 0 lvm /home └─centos-var_log 252:5 0 10G 0 lvm /var/log In addition to the above multipathd configuration, we have use_devicesfile=1 in /etc/lvm/lvm.conf, with /etc/lvm/devices/system.devices file being like so, with PVID used from the output of the pvdisplay command, and the IDNAME value extracted from the ouput of "ls -lha /dev/disk/by-id": VERSION=1.1.1 IDTYPE=sys_wwid IDNAME=eui.<drive_id> DEVNAME=/dev/nvme0n1p3 PVID=<pvid> PART=3 IDTYPE=sys_wwid IDNAME=naa.<drive_id> DEVNAME=/dev/sda PVID=<pvid> Issues started when performing certain tests regarding the system's integrity, most important of which is emergency shutdown's and reboot of all the nodes, the behavior that follows is that the OSDs are not started automatically as well as their respective LVM volumes not properly showing (except on a single node for some reason), hence the lsblk ouput changes like the snippet below, requiring us rebooting the nodes one by one until all the OSDs are back online: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 894.3G 0 disk nvme0n1 259:0 0 238.5G 0 disk ├─nvme0n1p1 259:1 0 600M 0 part /boot/efi ├─nvme0n1p2 259:2 0 1G 0 part /boot └─nvme0n1p3 259:3 0 236.9G 0 part ├─centos-root 252:0 0 170G 0 lvm / ├─centos-swap 252:1 0 23.4G 0 lvm [SWAP] ├─centos-var_log_audit 252:2 0 7.5G 0 lvm /var/log/audit ├─centos-home 252:4 0 26G 0 lvm /home └─centos-var_log 252:5 0 10G 0 lvm /var/log Without the LVM configuration and the multipathd service enabled, everything works fine, this behavior started happening after the changes. Attempting to manually restart the OSDs from a manager node using ceph orch daemon restart osd.n results in an error state, and even when manually starting the OSD on each node via the bash /var/lib/ceph/<fsid>/osd.0/unit.run we receive the following error: --> Failed to activate via raw: did not find any matching OSD to activate Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-<id>/osd-block-<block_id> --path /var/lib/ceph/osd/ceph-0 --no-mon-config stderr: failed to read label for /dev/ceph-<id>/osd-block-<block_id>: (2) No such file or directory 2024-03-30T12:42:54.014+0000 7f845296a980 -1 bluestore(/dev/ceph-<id>/osd-block-<block_id>) _read_bdev_label failed to open /dev/ceph-<id>/osd-block-<block_id>: (2) No such file or directory --> Failed to activate via LVM: command returned non-zero exit status: 1 --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' --> Failed to activate any OSD(s) A successful run of the same command but with success results in the following output: /bin/bash /var/lib/ceph/<fsid>/osd.0/unit.run Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/ceph--<id>-osd--block--<block_id> Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/ceph--<id>-osd--block--<block_id> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Running command: /usr/bin/ln -s /dev/mapper/ceph--<id>-osd--block--<block_id> /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 --> ceph-volume raw activate successful for osd ID: 0 ceph-<fsid>-osd-0 4361e2f166bcdeee6e9020dcbb153d3d7eec04e71d5b0b250440d4a3a0833f2c It seems to us as if the logical volume in cases of failure is not even detected at boot by the device-mapper, which is weird, it's also not showing in the output of the dmsetup ls command in cases of failure. What could we be missing here? What seems to be the conflict between CEPH OSDs and the multipathd service or even the LVM configuration? Should the system.devices entry be different than what we set? Is the multipathd blacklisting configuration missing something? We have been working on trial and error experiments for more than a week now and looked at the lvm2 as well as multipathd logs (we could provide them upon request) but to no avail as nothing indicates any errors, just normal logs with the difference being the missing CEPH OSD LVM volume. Best regards

1 0

cephfs inode backtrace information
by Dietmar Rieder 02 Apr '24

02 Apr '24

Hello, I have a question regarding the default pool of a cephfs. According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information? Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data). Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool? Thanks Dietmar

6 10

Drained A Single Node Host On Accident
by adam.ther 02 Apr '24

02 Apr '24

Hello, I have a single node host with a VM as a backup MON,MGR,ect. This has caused all OSD's to be pending as 'deleting', can i safely cancel this deletion request? Regards, Adam

2 1

S3 Partial Reads from Erasure Pool
by E3GH75＠gmail.com 01 Apr '24

01 Apr '24

I am dealing with a cluster that is having terrible performance with partial reads from an erasure coded pool. Warp tests and s3bench tests result in acceptable performance but when the application hits the data, performance plummets. Can anyone clear this up for me, When radosgw gets a partial read does it have to assemble all the rados objects that make up the s3 object before returning the range? With a replicated poll i am seeing 6 to 7 GiB/s of read performance and only 1GiB/s of read from the erasure coded pool which leads me to believe that the replicated pool is returning just the rados objects for the partial s3 object and the erasure coded pool is not.

2 1

Can setting mds_session_blocklist_on_timeout to false minize the session eviction?
by Yongseok Oh 01 Apr '24

01 Apr '24

Hi, CephFS is provided as a shared file system service in a private cloud environment of our company, LINE. The number of sessions is approximately more than 5,000, and session evictions occur several times a day. When session eviction occurs, the message 'Cannot send after transport endpoint shutdown' or 'Permission denied' is displayed and file system access is not possible. Our users are very uncomfortable with this issue. In particular, there are no special problems such as network connection or CPU usage. When I access the machine and take a close look, there are no special problems. After this, users feel the inconvenience of having to perform umount/mount tasks and run the application again. In a Kubernetes environment, recovery is a bit more complicated, which causes a lot of frustration. I tested that by setting the mds_session_blocklist_on_timeout and mds_session_blocklist_on_evict options to false and setting client_reconnect_stale to true on the client side, the file system can be accessed even if eviction occurs. It seemed like there was no major problem accessing the file system as the session was still attached. What I'm curious about is if I turn on the above option, will there be any other side effects? For example, should I take some action if the integrity of the file is broken or if there is an issue on the mds side? I am asking a question because there are no details regarding this in the official CephFS documentation. Thank you Yongseok

2 2

recreating a cephfs subvolume with the same absolute path
by Adiga, Anantha 29 Mar '24

29 Mar '24

Hi, ceph fs subvolume getpath cephfs cluster_A_subvolume cephfs_data_pool_ec21_subvolumegroup /volumes/cephfs_data_pool_ec21_subvolumegroup/cluster_A_subvolume/0f90806d-0d70-4fe1-9e2b-f958056ef0c9 If the subvolume got deleted, is it possible to recreate the subvolume with the same absolute path? so that yml specs that use the volume paths need not change Thank you, Anantha

1 0

v17.2.7 Quincy now supports Ubuntu 22.04 (Jammy Jellyfish)
by Casey Bodley 29 Mar '24

29 Mar '24

Ubuntu 22.04 packages are now available for the 17.2.7 Quincy release. The upcoming Squid release will not support Ubuntu 20.04 (Focal Fossa). Ubuntu users planning to upgrade from Quincy to Squid will first need to perform a distro upgrade to 22.04. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: b12291d110049b2f35e32e0de30d70e9a4c060d2

1 0

Improving CephFS performance by always putting "default" data pool on SSDs?
by Niklas Hambüchen 28 Mar '24

28 Mar '24

https://docs.ceph.com/en/reef/cephfs/createfs/ says: > The data pool used to create the file system is the “default” data pool and the location for storing all inode backtrace information, which is used for hard link management and disaster recovery. > For this reason, all CephFS inodes have at least one object in the default data pool. If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. This poses the question: Are normal replicated CephFS installations (metadata on SSDs, data on HDDs) set up with suboptimal performance because they don't do this? If having inodes/backtraces on replicated instead of EC improves performance, shouldn't one expect that putting inodes/backtraces on SSD would improve it even more? From the docs I also cannot really conclude when inotes/backtraces become important. Is that all the time, or only sometimes? Thanks!

2 3