- ceph-users - lists.ceph.io

Re: Syncing Error - (4) Incoming failed! ceph-users@ceph.io
by ceph.io 30 Sep '24

30 Sep '24

1 0

Undeliverable: Incoming Messages Failure
by Postmaster 29 Sep '24

29 Sep '24

1 0

RGW Graphs in cephadm setup
by brentk＠cfl.rr.com 29 Sep '24

29 Sep '24

We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of the clusters show the rgw metrics in the ceph dashboard and the other two don't. I made sure the firewalls were open for ceph-exporter and that Prometheus was gathering the stats on all 4 clusters. For the clusters that are not working, when I do a search on the Prometheus web console at port 9095 on the performance monitoring node using "{instance="rgw1"}, I noticed that the instance_id shows a long number. I did the same search on the clusters that show rgw stats in grafana and they have an instance_id that looks like the drop down in the Grafana dashboard. E.g. "rgw.rgw.rgw1.biewae". So it seems that for some reason, on the clusters that are not working, the ceph-exporter running on the rgw nodes is not pulling the proper instance_id rgw instance. Is there a way to fix the instance_id that ceph-exporter is pulling? I can see this instance id when browsing to the servers http://rgw1:9926/metrics portal. If I run a podman ps, I can see the container running and it has the unique id .biewae showing. I used cephadm to enter the ceph-exporter container on the rgw to see what the logs were saying and I see the same thing repeated: 1 pid path is empty; process metrics won't be fetched for: ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864 Its only the rgw's that show this error in the ceph-exporter log file, the mons and osd nodes show no errors in the log and show up in grafana properly with data. I redeployed ceph-exporter and rgw to the same node in that order, but no change. I did set the default image container to 18.2.4 prior to redeploy cause I ran into that issue as well. Regards, -Brent Existing Clusters: Test: Reef 18.2.0 ( all virtual on nvme ) US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw, 2 iscsigw UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw, 2 iscsigw US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw

1 0

Old MDS container version when: Ceph orch apply mds
by opositorvlc＠mail.com 28 Sep '24

28 Sep '24

Hi All, I migrated my CEPH 18.2.2 cluster from a non cephadm configuration. All goes fine except MDS service was deployed in a old version: 17.0.0 I'm trying to deploy MDS daemons using ceph orch but CEPH always download an old MDS image from docker. How could I deploy the MDS service in the same 18.2.2 version that the rest of services? [root@master1 ~]# ceph orch apply mds datafs --placement="2 master1 master2" [root@master1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.datafs.master1.gcpovr master1 running (36m) 6m ago 36m 37.2M - 17.0.0-7183-g54142666 75e3d7089cea 96682779c7ad mds.datafs.master2.oqaxuy master2 running (36m) 6m ago 36m 33.1M - 17.0.0-7183-g54142666 75e3d7089cea a9a647f87c83 mgr.master master1 running (16h) 6m ago 17h 448M - 18.2.2 3c937764e6f5 70f06fa05b70 mgr.master2 master2 running (16h) 6m ago 17h 524M - 18.2.2 3c937764e6f5 2d0d5376d8b3 mon.master master1 running (16h) 6m ago 17h 384M 2048M 18.2.2 3c937764e6f5 66a65017ce29 mon.master2 master2 running (16h) 6m ago 17h 380M 2048M 18.2.2 3c937764e6f5 51d783a9e36c osd.0 osd00 running (16h) 3m ago 17h 432M 4096M 18.2.2 3c937764e6f5 fedff66f5ed2 osd.1 osd00 running (16h) 3m ago 17h 475M 4096M 18.2.2 3c937764e6f5 24e24a1a22e6 osd.2 osd00 running (16h) 3m ago 17h 516M 4096M 18.2.2 3c937764e6f5 ccd05451b739 osd.3 osd00 running (16h) 3m ago 17h 454M 4096M 18.2.2 3c937764e6f5 f6d8f13c8aaf osd.4 master1 running (16h) 6m ago 17h 525M 4096M 18.2.2 3c937764e6f5 a2dcf9f1a9b7 osd.5 master2 running (16h) 6m ago 17h 331M 4096M 18.2.2 3c937764e6f5 b0011e8561a4 [root@master1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.datafs 2/2 6m ago 46s master1;master2;count:2 mgr 2/0 6m ago - <unmanaged> mon 2/0 6m ago - <unmanaged> osd 6 6m ago - <unmanaged> [root@master1 ~]# ceph versions { "mon": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "mgr": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "osd": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 6 }, "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2 }, "overall": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 10 } } [root@master1 ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.28 GB quay.io/ceph/ceph v18 3c937764e6f5 7 weeks ago 1.28 GB registry.access.redhat.com/ubi8 latest c70d72aaebb4 3 months ago 212 MB quay.io/ceph/ceph v16 0d668911f040 23 months ago 1.27 GB quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years ago 571 MB quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years ago 205 MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years ago 22.3 MB quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 2 years ago 58.9 MB docker.io/ceph/daemon-base latest-master-devel 75e3d7089cea 2 years ago 1.29 GB

3 2

17.2.8 release date?
by Szabo, Istvan (Agoda) 28 Sep '24

28 Sep '24

Hi, Do we know roughly when the 17.2.8 Quincy is going to be released? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

v19.2.0 Squid released
by Laura Flores 28 Sep '24

28 Sep '24

We're very happy to announce the first stable release of the Squid series. We express our gratitude to all members of the Ceph community who contributed by proposing pull requests, testing this release, providing feedback, and offering valuable suggestions. Highlights: RADOS * BlueStore has been optimized for better performance in snapshot-intensive workloads. * BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. * Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Improved navigation layout CephFS * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RBD * diff-iterate can now execute locally, bringing a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * Support for cloning from non-user type snapshots is added. * rbd-wnbd driver has gained the ability to multiplex image mappings. RGW * The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. Crimson/Seastore * Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson We encourage you to read the full release notes at https://ceph.io/en/news/blog/2024/v19-2-0-squid-released/ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 16063ff2022298c9300e49a547a16ffda59baf13 -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 2

WAL on NVMe/SSD not used after OSD/HDD replace
by mailing-lists 28 Sep '24

28 Sep '24

Dear Ceph-users, I have a problem that I'd like to have your input for. Preface: I have got a test-cluster and a productive-cluster. Both are setup the same and both are having the same "issue". I am running Ubuntu 22.04 and deployed ceph 17.2.3 via cephadm. Upgraded to 17.2.7 later on, which is the version we are currently running. Since the issue seem to be the exact same on the test-cluster, I will post test-cluster-outputs here for better readability. The issue: I have replaced disks and after the replacement, it does not show that it would use the NVMe as WAL device anymore. The LV still exists, but the metadata of the osd does not show it, as it would be with any other osd/hdd, that hasnt been replaced. ODS.1 (incorrect, bluefs_dedicated_wal: "0") ``` { "id": 1, "arch": "x86_64", "back_addr": "[v2:192.168.6.241:6802/3213655489,v1:192.168.6.241:6803/3213655489]", "back_iface": "", "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "0", "bluefs_single_shared_device": "1", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-3", "bluestore_bdev_devices": "sdd", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_optimal_io_size": "0", "bluestore_bdev_partition_path": "/dev/dm-3", "bluestore_bdev_rotational": "1", "bluestore_bdev_size": "17175674880", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "hdd", "bluestore_min_alloc_size": "4096", "ceph_release": "quincy", "ceph_version": "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)", "ceph_version_short": "17.2.7", "ceph_version_when_created": "", "container_hostname": "bi-ubu-srv-ceph2-01", "container_image": "quay.io/ceph/ceph@sha256:28323e41a7d17db238bdcc0a4d7f38d272f75c1a499bc30f59b0b504af132c6b", "cpu": "AMD EPYC 75F3 32-Core Processor", "created_at": "", "default_device_class": "hdd", "device_ids": "sdd=QEMU_HARDDISK_drive-scsi3", "device_paths": "sdd=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:3:0", "devices": "sdd", "distro": "centos", "distro_description": "CentOS Stream 8", "distro_version": "8", "front_addr": "[v2:<my_public_ip>.241:6800/3213655489,v1:<my_public_ip>.241:6801/3213655489]", "front_iface": "", "hb_back_addr": "[v2:192.168.6.241:6806/3213655489,v1:192.168.6.241:6807/3213655489]", "hb_front_addr": "[v2:<my_public_ip>.241:6804/3213655489,v1:<my_public_ip>.241:6805/3213655489]", "hostname": "bi-ubu-srv-ceph2-01", "journal_rotational": "1", "kernel_description": "#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024", "kernel_version": "5.15.0-122-generic", "mem_swap_kb": "4018172", "mem_total_kb": "5025288", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_unknown_devices": "sdd", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-1", "osd_objectstore": "bluestore", "osdspec_affinity": "dashboard-admin-1661853488642", "rotational": "1" } ``` ODS.0 (correct, bluefs_dedicated_wal: "1") ``` { "id": 0, "arch": "x86_64", "back_addr": "[v2:192.168.6.241:6810/3249286142,v1:192.168.6.241:6811/3249286142]", "back_iface": "", "bluefs": "1", "bluefs_dedicated_db": "0", "bluefs_dedicated_wal": "1", "bluefs_single_shared_device": "0", "bluefs_wal_access_mode": "blk", "bluefs_wal_block_size": "4096", "bluefs_wal_dev_node": "/dev/dm-0", "bluefs_wal_devices": "sdb", "bluefs_wal_driver": "KernelDevice", "bluefs_wal_optimal_io_size": "0", "bluefs_wal_partition_path": "/dev/dm-0", "bluefs_wal_rotational": "0", "bluefs_wal_size": "4290772992", "bluefs_wal_support_discard": "1", "bluefs_wal_type": "ssd", "bluestore_bdev_access_mode": "blk", "bluestore_bdev_block_size": "4096", "bluestore_bdev_dev_node": "/dev/dm-2", "bluestore_bdev_devices": "sdc", "bluestore_bdev_driver": "KernelDevice", "bluestore_bdev_optimal_io_size": "0", "bluestore_bdev_partition_path": "/dev/dm-2", "bluestore_bdev_rotational": "1", "bluestore_bdev_size": "17175674880", "bluestore_bdev_support_discard": "1", "bluestore_bdev_type": "hdd", "bluestore_min_alloc_size": "4096", "ceph_release": "quincy", "ceph_version": "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)", "ceph_version_short": "17.2.7", "ceph_version_when_created": "", "container_hostname": "bi-ubu-srv-ceph2-01", "container_image": "quay.io/ceph/ceph@sha256:28323e41a7d17db238bdcc0a4d7f38d272f75c1a499bc30f59b0b504af132c6b", "cpu": "AMD EPYC 75F3 32-Core Processor", "created_at": "", "default_device_class": "hdd", "device_ids": "sdb=QEMU_HARDDISK_drive-scsi1,sdc=QEMU_HARDDISK_drive-scsi2", "device_paths": "sdb=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:1:0,sdc=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:2:0", "devices": "sdb,sdc", "distro": "centos", "distro_description": "CentOS Stream 8", "distro_version": "8", "front_addr": "[v2:<my_public_ip>.241:6808/3249286142,v1:<my_public_ip>.241:6809/3249286142]", "front_iface": "", "hb_back_addr": "[v2:192.168.6.241:6814/3249286142,v1:192.168.6.241:6815/3249286142]", "hb_front_addr": "[v2:<my_public_ip>.241:6812/3249286142,v1:<my_public_ip>.241:6813/3249286142]", "hostname": "bi-ubu-srv-ceph2-01", "journal_rotational": "0", "kernel_description": "#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024", "kernel_version": "5.15.0-122-generic", "mem_swap_kb": "4018172", "mem_total_kb": "5025288", "network_numa_unknown_ifaces": "back_iface,front_iface", "objectstore_numa_unknown_devices": "sdb,sdc", "os": "Linux", "osd_data": "/var/lib/ceph/osd/ceph-0", "osd_objectstore": "bluestore", "osdspec_affinity": "dashboard-admin-1661853488642", "rotational": "1" } ``` The LV that should contain the WAL is existing: ``` :/# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368 -wi-ao---- <16.00g osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35 -wi-ao---- <16.00g osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd ceph-a84cbc55-33b5-4c31-8426-66e375fc157b -wi-ao---- <4.00g osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431 ceph-a84cbc55-33b5-4c31-8426-66e375fc157b -wi-a----- <4.00g ``` We can also see there, that there is no (o) as in open in the LV attributes. For good measure, this is the pattern for the orch osd service. ``` service_type: osd service_id: dashboard-admin-1661853488642 service_name: osd.dashboard-admin-1661853488642 placement: host_pattern: '*' spec: data_devices: size: 16GB db_devices: rotational: false filter_logic: AND objectstore: bluestore wal_devices: rotational: false status: created: '2022-08-30T09:58:11.210571Z' last_refresh: '2024-09-27T12:19:14.619434Z' running: 16 size: 16 ``` "ceph-volume lvm list" looks fine, in my opinion, as it should be: ``` { "0": [ { "devices": [ "/dev/sdc" ], "lv_name": "osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "lv_path": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "lv_size": "17175674880", "lv_tags": "ceph.block_device=/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.block_uuid=3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.osd_id=0,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=block,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd,ceph.wal_uuid=BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et", "lv_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk", "name": "osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "path": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "tags": { "ceph.block_device": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.block_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.osd_id": "0", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "block", "ceph.vdo": "0", "ceph.wal_device": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "ceph.wal_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et" }, "type": "block", "vg_name": "ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368" }, { "devices": [ "/dev/sdb" ], "lv_name": "osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "lv_path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "lv_size": "4290772992", "lv_tags": "ceph.block_device=/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.block_uuid=3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.osd_id=0,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd,ceph.wal_uuid=BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et", "lv_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et", "name": "osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "tags": { "ceph.block_device": "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.block_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "d472bf9f-c17d-4939-baf5-514a07db66bc", "ceph.osd_id": "0", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "wal", "ceph.vdo": "0", "ceph.wal_device": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd", "ceph.wal_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et" }, "type": "wal", "vg_name": "ceph-a84cbc55-33b5-4c31-8426-66e375fc157b" } ], "1": [ { "devices": [ "/dev/sdd" ], "lv_name": "osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "lv_path": "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "lv_size": "17175674880", "lv_tags": "ceph.block_device=/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f,ceph.block_uuid=QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d94bda82-e59f-4d3d-81cd-28ea69c5e02f,ceph.osd_id=1,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=block,ceph.vdo=0", "lv_uuid": "QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO", "name": "osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "path": "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "tags": { "ceph.block_device": "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "ceph.block_uuid": "QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "d94bda82-e59f-4d3d-81cd-28ea69c5e02f", "ceph.osd_id": "1", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "block", "ceph.vdo": "0" }, "type": "block", "vg_name": "ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35" }, { "devices": [ "/dev/sdb" ], "lv_name": "osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "lv_path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "lv_size": "4290772992", "lv_tags": "ceph.block_device=/dev/ceph-179656b5-6783-4b4e-bb7e-695d4523072f/osd-block-7a1d0007-71ff-4011-8a18-e6de1499cbdf,ceph.block_uuid=gDt63k-v0Qd-89WE-bOGO-cmZv-Sc8v-tYOagJ,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=7a1d0007-71ff-4011-8a18-e6de1499cbdf,ceph.osd_id=1,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431,ceph.wal_uuid=UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq", "lv_uuid": "UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq", "name": "osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "path": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "tags": { "ceph.block_device": "/dev/ceph-179656b5-6783-4b4e-bb7e-695d4523072f/osd-block-7a1d0007-71ff-4011-8a18-e6de1499cbdf", "ceph.block_uuid": "gDt63k-v0Qd-89WE-bOGO-cmZv-Sc8v-tYOagJ", "ceph.cephx_lockbox_secret": "", "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a", "ceph.cluster_name": "ceph", "ceph.crush_device_class": "", "ceph.encrypted": "0", "ceph.osd_fsid": "7a1d0007-71ff-4011-8a18-e6de1499cbdf", "ceph.osd_id": "1", "ceph.osdspec_affinity": "dashboard-admin-1661853488642", "ceph.type": "wal", "ceph.vdo": "0", "ceph.wal_device": "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431", "ceph.wal_uuid": "UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq" }, "type": "wal", "vg_name": "ceph-a84cbc55-33b5-4c31-8426-66e375fc157b" } ] } ``` We can see two devices for osd.1, but the osd does not seem to use the WAL device. How can i tell osd.1 to use its WAL again? Why does the orch osd service not do it anyways? When i was replacing the disks i was using ``` ceph orch osd rm 1 --replace ``` The osd got back up and refilled, but without WAL. While I was diggin into this, i realized that the DB is not on the NVME, but on the HDD itself. Even though, i was specifying it in the pattern above: db_devices: rotational: false Shouldnt the db then be located on the NVMe/SSD where the wal is at? At least that is what the dashboard suggest, when creating OSDs. Of course I chose the NVME/SSD for DB and WAL. I hope this is somewhat readable and coherent. Thank you in advance! Best wishes and a nice weekend!

4 3

Mds daemon damaged - assert failed
by Kyriazis, George 27 Sep '24

27 Sep '24

Hello ceph users, I am in the unfortunate situation of having a status of “1 mds daemon damaged”. Looking at the logs, I see that the daemon died with an assert as follows: ./src/osdc/Journaler.cc: 1368: FAILED ceph_assert(trim_to > trimming_pos) ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12a) [0x73a83189d7d9] 2: /usr/lib/ceph/libceph-common.so.2(+0x29d974) [0x73a83189d974] 3: (Journaler::_trim()+0x671) [0x57235caa70b1] 4: (Journaler::_finish_write_head(int, Journaler::Header&, C_OnFinisher*)+0x171) [0x57235caaa8f1] 5: (Context::complete(int)+0x9) [0x57235c716849] 6: (Finisher::finisher_thread_entry()+0x16d) [0x73a83194659d] 7: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x73a8310a8134] 8: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x73a8311287dc] 0> 2024-09-23T14:10:26.490-0500 73a822c006c0 -1 *** Caught signal (Aborted) ** in thread 73a822c006c0 thread_name:MR_Finisher ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x73a83105b050] 2: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x73a8310a9e2c] 3: gsignal() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x185) [0x73a83189d834] 6: /usr/lib/ceph/libceph-common.so.2(+0x29d974) [0x73a83189d974] 7: (Journaler::_trim()+0x671) [0x57235caa70b1] 8: (Journaler::_finish_write_head(int, Journaler::Header&, C_OnFinisher*)+0x171) [0x57235caaa8f1] 9: (Context::complete(int)+0x9) [0x57235c716849] 10: (Finisher::finisher_thread_entry()+0x16d) [0x73a83194659d] 11: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x73a8310a8134] 12: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x73a8311287dc] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. As listed above, I am running 18.2.2 on a proxmox cluster with a hybrid hdd/sdd setup. 2 cephfs filesystems. The mds responsible for the hdd filesystem is the one that died. Output of ceph -s follows: root@vis-mgmt:~/bin# ceph -s cluster: id: ec2c9542-dc1b-4af6-9f21-0adbcabb9452 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 5 pgs not scrubbed in time 1 daemons have recently crashed services: mon: 5 daemons, quorum vis-hsw-01,vis-skx-01,vis-clx-15,vis-clx-04,vis-icx-00 (age 6m) mgr: vis-hsw-02(active, since 13d), standbys: vis-skx-02, vis-hsw-04, vis-clx-08, vis-clx-02 mds: 1/2 daemons up, 5 standby osd: 97 osds: 97 up (since 3h), 97 in (since 4d) data: volumes: 1/2 healthy, 1 recovering; 1 damaged pools: 14 pools, 1961 pgs objects: 223.70M objects, 304 TiB usage: 805 TiB used, 383 TiB / 1.2 PiB avail pgs: 1948 active+clean 9 active+clean+scrubbing+deep 4 active+clean+scrubbing io: client: 86 KiB/s rd, 5.5 MiB/s wr, 64 op/s rd, 26 op/s wr I tried restarting all the mds deamons but they are all marked as “standby”. I also tried restarting all the mons and then the mds daemons again, but that didn’t help. Much help is appreciated! Thank you! George

4 8

All monitors fall down simultaneously when I try to map rbd on client
by Alex from North 27 Sep '24

27 Sep '24

Hello everybody, found intresting thing: for some reason ALL the monitors crash when I try to rbd map on client host. here is my pool: root@ceph1:~# ceph osd pool ls iotest Here is my rbd in this pool: root@ceph1:~# rbd ls -p iotest test1 this is a client creds to connect to this pool: [client.iotest] key = AQASVfZm5bPGLBAAXyPWqJvNMBsXsJQcFrSAhg== caps mgr = "profile rbd pool=iotest" caps mon = "profile rbd" caps osd = "profile rbd pool=iotest" This is rbmap file on a client host: root@node-stat:/etc/ceph# cat rbdmap # RbdDevice Parameters #poolname/imagename id=client,keyring=/etc/ceph/ceph.client.keyring iotest/test1 id=iotest,keyring=/etc/ceph/ceph.client.iotest.keyring So, when I press Enter on command rbd map iotest/test1 --id iotest in the same moment ALL the mons go down. I pul log on pastebin as it is quite long https://pastebin.com/iCr8pY1r All the hints are appreciated. Thanks in advance.

3 5

Restore a pool from snapshot
by Pavel Kaygorodov 27 Sep '24

27 Sep '24

Hi! May be a dumb question, sorry, but how I can restore a whole pool from a snapshot? I have made a snapshot with 'rados mksnap', but there is no command to restore a whole snapshot, only one object may be specified for rollback. Is it possible to restore all objects? Thanks in advance, Pavel.

2 1