Please don't drop the list from your responses. Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that. Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
Hi,
I would expect that you have a similar config-key entry:
ceph config-key ls |grep "peer/cephfs" "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56",
Maybe removing that peer would already suffice?
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software.
The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication ', but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ?
Best regards, Jan Zeinstra _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io