- ceph-users - lists.ceph.io

Safe method to perform failback for RBD on one way mirroring.
by Saif Mohammad 28 May '24

28 May '24

Hello Everyone We have Clusters in production with the following configuration: Cluster-A : quincy v17.2.5 Cluster-B : quincy v17.2.5 All images in a pool have the snapshot feature enabled and are mirrored. Each site has 3 daemons. We're testing disaster recovery with one-way mirroring in our block device mirroring setup. On the primary site (Cluster-A), we have Ceph clients attached to it , and couple of images are present there. These images are replicated to the secondary site (Cluster-B). During testing we've successfully conducted failovers, with all resources accessible from Cluster-B once the Ceph client is attached there on secondary site. However, during failback (restoring the primary site), we've encountered an issue. Data that was pushed from the secondary site seems to be deleted, while data originally present only on the primary site remains intact. Here are the steps we took during failback: - Detached the client from Cluster-B. - Ensured that on Cluster-B, "mirroring primary" is set to true, and on Cluster-A, it's set to false. - Demoted the images on Cluster-B and promoted the images on Cluster-A. After performing these steps, our images went into an error state, and they started syncing from Cluster-A to Cluster-B. However, in a failback scenario, the direction should be from Cluster-B to Cluster-A. We are not sure where we are making mistakes. Could anybody please advise on the correct procedure for failback in one-way mirroring and the safest way to execute it to avoid impacting our data. Regards, Mohammad Saif

2 1

does the RBD client block write when the Watcher times out?
by Yuma Ogami 28 May '24

28 May '24

Hello. I'm currently verifying the behavior of RBD on failure. I'm wondering about the consistency of RBD images after network failures. As a result of my investigation, I found that RBD sets a Watcher to RBD image if a client mounts this volume to prevent multiple mounts. In addition, I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario? I tested the following case on which the watcher was released by hand and detected data corruption. 1. Release the Watcher on the node (A) that mounts the RBD using the `ceph osd blocklist add` command 2. Another node (B) mounts the RBD volume. 3. Unblock using the `ceph osd blocklist rm` command 4. Write from node (B) (write successfully) 5. Write from node (A) (can be written successfully from the application's point of view. In fact, the write fails) 6. Write content at node (A) is lost. In this case, I released the watcher by hand to emulate the timeout due to network failure. It's because I couldn't emulate real network failure in this test. I considered using exclusive lock and restricting access to those from a single node. However, we gave up on that as blocking writes entirely would make snapshots non-functional. The version of Ceph we are using is v17.2.6. Best regards, Yuma.

4 4

Ceph ARM providing storage for x86
by filip Mutterer 27 May '24

27 May '24

Is this known to be working: Setting up the Ceph Cluster with ARM and then use the storage with X86 Machines for example LXC, Docker and KVM? Is this possible? Greetings filip

3 2

RBD Mirror - Failed to unlink peer
by scott.cairns＠tecnica-ltd.co.uk 27 May '24

27 May '24

I've had rbd mirror deployed for over a year copying images from Site A to Site B for DR. After introducing an additional node into the cluster a few weeks back and creating OSD's on this node (no mon/mgr) we've started to see snapshots throwing an error 'librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f24fc001bf0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory' However, the snapshot still completes (provides a Snapshot ID) and RBD mirror still pulls the snapshot on the remote end, so it appears to be working as expected, however every snapshot is generating this error. I can't see anything else logged when this error is generated, it's progressively got worse since the start (the first day we saw around 2 errors on 150 snapshots, it's now about 145 errors on 150 snapshots) The only other reference I can see to this is from 2 years ago and was related to a bug which was fixed a while ago. I'm running Ceph 17.2.7. Any idea what file/directory it's having issues with? Thanks.

1 0

RBD Mirror - implicit snapshot cleanup
by scott.cairns＠tecnica-ltd.co.uk 27 May '24

27 May '24

With RBD Mirroring configured from Site A to Site B, we typically see 2 snapshots on the source side (current and 1 previous) and 1 snapshot on the destination side (most recent snapshot from source) - this has been the case for over a year now. When testing a DR scenario (i.e. demoting the primary, promoting the secondary, testing, resyncing, demoting the secondary, promoting the primary) we then result in a lot more snapshots on each side - currently around 6 on the source side (demoted image, then current and 4 previous) and 4 on the destination side (original snapshot, then snapshot from the promote, demote, then the current snapshot from source). Is it possible to clear these existing snapshots, or is there some sort of schedule that'll automatically purge them, so that we return to 2 snapshots on source and 1 snapshot on destination? Thanks.

1 0

ceph orch issue: lsblk: /dev/vg_osd/lvm_osd: not a block device
by duluxoz 27 May '24

27 May '24

Hi All, Is the following a bug or some other problem (I can't tell) :-) Brand new Ceph (Reef v18.2.3) install on Rocky Linux v9.4 - basically, its a brand new box. Ran the following commands (in order; no issues until final command): 1. pvcreate /dev/sda6 2. vgcreate vg_osd /dev/sda6 3. lvcreate -l 100%VG -n lv_osd vg_osd 4. cephadmbootstrap--mon-ip192.168.0.20 5. ceph orch daemon add osd ceph1:/dev/vg_osd/lvm_osd Received a whole bunch of error info on the console; the two relevant lines (as far as I can tell) are: * /usr/bin/podman: stderr stderr: lsblk: /dev/vg_osd/lvm_osd: not a block device * RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:257b3f5140c11b51fd710ffdad6213ed53d74146f464a51717262d156daef553 -e NODE_NAME=ceph1 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=None -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/477045f4-1b34-11ef-9a30-0800274c7359:/var/run/ceph:z -v /var/log/ceph/477045f4-1b34-11ef-9a30-0800274c7359:/var/log/ceph:z -v /var/lib/ceph/477045f4-1b34-11ef-9a30-0800274c7359/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/477045f4-1b34-11ef-9a30-0800274c7359/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpe_krhtt8:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_47jsxdp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:257b3f5140c11b51fd710ffdad6213ed53d74146f464a51717262d156daef553 lvm batch --no-auto /dev/vg_osd/lvm_osd --yes --no-systemd I had a look around the Net and couldn't find anything relevant. This post (https://github.com/rook/rook/issues/4967) talks about a similar issue using Rook, but I'm not using Rook but cephadm. Any help in resolving this (or confirming it is a bug) would be greatly appreciated - thanks in advance. Cheers Dulux-Oz

4 6

Problem in changing monitor address and public_network
by farhad kh 27 May '24

27 May '24

Hello, according to ceph own document and the article that I sent the link to, I tried to change the address of the ceph machines and its public network. But the guarantee that I had to set the machines with the new address(ceph orch host set-addr opcrgfpsksa0101 10.248.35.213) , the command was not executed and the managers were trying to connect to the same previous addresses. I have run my cluster with a non-root user and i change adress from 10.56.12.0/22 to 10.248.35/24. i used ceph version 17.2.6 and cephadm. Here, I don't have a configuration for the cluster_network and all communications are on the public_network. Also, I have sent the specifications and logs that I think are useful Are there any ideas and solutions to fix this problem and the cause of it? -------------- debug 2024-05-26T17:28:33.191+0000 7ffb15579700 0 log_channel(cephadm) log [INF] : Filtered out host opcrgfpsksa0101: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.191+0000 7ffb15579700 0 [cephadm INFO cephadm.serve] Filtered out host opcrgfpsksa0103: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.191+0000 7ffb15579700 0 log_channel(cephadm) log [INF] : Filtered out host opcrgfpsksa0103: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 [cephadm INFO cephadm.serve] Filtered out host opcmrfpsksa0101: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 log_channel(cephadm) log [INF] : Filtered out host opcmrfpsksa0101: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 [progress WARNING root] complete: ev da5c20ec-e9df-490f-804d-182d02f0324e does not exist debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 [progress WARNING root] complete: ev 82392bf5-4940-4252-9aaf-aa4758c00ead does not exist debug 2024-05-26T17:28:33.212+0000 7ffb15579700 0 [progress WARNING root] complete: ev 1d64774f-dc6d-46c8-8f96-21d147f4b053 does not exist debug 2024-05-26T17:28:33.212+0000 7ffb15579700 0 [progress WARNING root] complete: ev 45322c73-8ecb-4471-b41c-0e279805dd0b does not exist debug 2024-05-26T17:28:35.182+0000 7ffb584e1700 0 log_channel(cluster) log [DBG] : pgmap v232: 1185 pgs: 1185 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2024-05-26T17:28:37.183+0000 7ffb584e1700 0 log_channel(cluster) log [DBG] : pgmap v233: 1185 pgs: 1185 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2024-05-26T17:28:38.609+0000 7ffb4f4df700 0 log_channel(audit) log [DBG] : from='client.14874149 -' entity='client.admin' cmd=[{"prefix": "orch host set-addr", "hostname": "opcrgfpsksa0101", "addr": "10.248.35.213", "target": ["mon-mgr", ""]}]: dispatch debug 2024-05-26T17:28:39.184+0000 7ffb584e1700 0 log_channel(cluster) log [DBG] : pgmap v234: 1185 pgs: 1185 unknown; 0 B data, 0 B used, 0 B / 0 B avail ------------ ceph orch host ls HOST ADDR LABELS STATUS opcmrfpsksa0101 10.56.12.216 _admin mon osd opcpmfpsksa0101 10.56.12.204 rgw opcpmfpsksa0103 10.56.12.205 rgw opcpmfpsksa0105 10.56.12.206 rgw opcrgfpsksa0101 10.56.12.213 _admin mon osd opcrgfpsksa0103 10.56.12.214 _admin mon osd opcsdfpsksa0101 10.56.12.207 osd opcsdfpsksa0103 10.56.12.208 osd opcsdfpsksa0105 10.56.12.209 osd 9 hosts in cluster ------------------------ NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 3/3 8m ago 11M count:3 ceph-exporter 9/9 8m ago 7w * crash 9/9 8m ago 11M * grafana ?:3000 3/3 8m ago 11d count:3;label:mon ingress.rgw.k8s 10.56.12.215:80,1967 4/4 8m ago 11d count:2;label:rgw mds.k8s-cephfs 3/3 8m ago 11M count:3 mgr 2/2 7m ago 11d count:2;label:mon mon 3/3 8m ago 7M count:3;label:mon node-exporter ?:9100 9/9 8m ago 7w * osd 8 8m ago - <unmanaged> osd.dashboard-admin-1695638488579 37 8m ago 8M * prometheus ?:9095 3/3 8m ago 7w count:3;label:mon rgw.k8s ?:8080 3/3 8m ago 5M count:3;label:rgw

1 0

Lousy recovery for mclock and reef
by Mazzystr 24 May '24

24 May '24

Hi all, Goodness I'd say it's been at least 3 major releases since I had to do a recovery. I have disks with 60-75,000 power_on_hours. I just updated from Octopus to Reef last month and I'm hit with 3 disk failures and the mclock ugliness. My recovery is moving at a wondrous 21 mb/sec after some serious hacking. It started out at 9 mb/sec. My hosts are showing minimal cpu use. normal mem use. 0-6% disk business. Load is minimal so processes aren't blocked by disk io. I tried the changing all the sleeps and recovery_max and setting osd_mclock_profile high_recovery_ops to no change in performance. Does anyone have any suggestions to improve performance? Thanks, /Chris C

2 5

Reef RGWs stop processing requests
by Iain Stott 24 May '24

24 May '24

Hi, We are running 3 clusters in multisite. All 3 were running Quincy 17.2.6 and using cephadm. We upgraded one of the secondary sites to Reef 18.2.1 a couple of weeks ago and were planning on doing the rest shortly afterwards. We run 3 RGW daemons on separate physical hosts behind an external HAProxy HA pair for each cluster. Since we upgrade to Reef we have had issues with the RGWs stopping processing requests. We can see that they don't crash as they still have entries in the logs about syncing, but as far as request processing goes, they just stop. While debugging this we have 1 of the 3 RGWs running a Quincy image, and this has never had an issue where it stops processing requests. Any Reef containers we deploy have always stopped within 48Hrs of being deployed. We have tried Reef versions 18.2.1, 18.2.2 and 18.1.3 and all exhibit the same issue. We are running podman 4.6.1 on Centos 8 with kernel 4.18.0-513.24.1.el8_9.x86_64. We have enabled debug logs for the RGWs but we have been unable to find anything in them that would shed light on the cause. We are just wondering if anyone had any ideas on what could be causing this or how to debug it further? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

2 2

Cephadm quincy 17.2.5 always shows slowops in all OSDs and Ceph orch stuck
by pahrialtkj＠gmail.com 24 May '24

24 May '24

Hi all, I'm running a quincy 17.2.5 ceph cluster, with 3 nodes that each have 3 disks with replica size 3 min_size 2, my cluster was running fine before, and suddenly it can't read and write because ceph slowops on all osd. I tried restarting osd, then degraded and misplaced data appeared, and slow ops reappeared. Any suggestions guys to solve this issues ? # ceph -s cluster: id: bb6590a9-155c-4830-aea7-ef4b5a098466 health: HEALTH_ERR 2 failed cephadm daemon(s) 1/282019 objects unfound (0.000%) noout flag(s) set Reduced data availability: 79 pgs inactive, 47 pgs peering Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 26340/838103 objects degraded (3.143%), 16 pgs degraded, 11 pgs undersized 302 slow ops, oldest one blocked for 226773 sec, daemons [osd.2,osd.3,osd.4,osd.6,osd.7] have slow ops. services: mon: 3 daemons, quorum r2c2,r2c3,r2c1 (age 2h) mgr: r2c3(active, since 15m), standbys: r2c2, r2c1.ivwibd osd: 9 osds: 9 up (since 9m), 9 in (since 3d); 30 remapped pgs flags noout data: pools: 12 pools, 329 pgs objects: 282.02k objects, 1.3 TiB usage: 4.9 TiB used, 94 TiB / 99 TiB avail pgs: 24.924% pgs not active 26340/838103 objects degraded (3.143%) 5411/838103 objects misplaced (0.646%) 1/282019 objects unfound (0.000%) 240 active+clean 31 peering 23 activating 16 remapped+peering 7 activating+undersized+degraded+remapped 2 activating+remapped 2 active+undersized+degraded+remapped+backfilling 2 active+recovering+degraded 2 activating+degraded 1 active+recovery_unfound+undersized+degraded+remapped 1 activating+degraded+remapped 1 active+clean+scrubbing+deep 1 active+recovery_wait+undersized+degraded+remapped io: client: 12 KiB/s rd, 1.3 KiB/s wr, 7 op/s rd, 1 op/s wr Thanks, Pahrial

1 0