Re: Cannot reinstate ceph fs mirror because i destroyed the ceph fs mirror peer/ target server
Please don't drop the list from your responses. Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that. Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
Hi,
I would expect that you have a similar config-key entry:
ceph config-key ls |grep "peer/cephfs" "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56",
Maybe removing that peer would already suffice?
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software.
The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication ', but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ?
Best regards, Jan Zeinstra _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work: site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored I assume that this should suffice to start from scratch, but maybe you can confirm. Zitat von Eugen Block <eblock@nde.ag>:
Please don't drop the list from your responses.
Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
Hi,
I would expect that you have a similar config-key entry:
ceph config-key ls |grep "peer/cephfs" "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56",
Maybe removing that peer would already suffice?
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software.
The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication ', but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ?
Best regards, Jan Zeinstra _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Eugen, This works: so in conclusion, the assumption is that removing the config with: ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" And disabling the fs: ceph fs snapshot mirror disable proofs yields: ceph fs snapshot mirror peer_list prodfs Error EINVAL: filesystem prodfs is not mirrored Reenabling prodfs mirrorring: ceph fs snapshot mirror enable prodfs ceph fs snapshot mirror peer_list prodfs yields: {} So: very grateful for your help, and happily proceeding with re-enabling the mirroring solution. Again: Thank you, Jan Zeinstra Op ma 14 apr 2025 om 10:29 schreef Eugen Block <eblock@nde.ag>:
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work:
site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored
I assume that this should suffice to start from scratch, but maybe you can confirm.
Zitat von Eugen Block <eblock@nde.ag>:
Please don't drop the list from your responses.
Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
Hi,
I would expect that you have a similar config-key entry:
ceph config-key ls |grep "peer/cephfs" "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56",
Maybe removing that peer would already suffice?
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software.
The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See
http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication ',
but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ?
Best regards, Jan Zeinstra _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Glad it works for you. May I ask, did you have difficulties setting up mirroring? I tried half a year ago and failed. Then I tried again last week, I succeeded, but I faced several issues along the way. I would like to improve the docs (I'm in contact with Zac about that), but if you didn't encounter any issues and everything was clear, maybe I'm the issue here. ;-) Anyway, if you could share your setup experience, I could compare and figure out which part of the docs could use some improvement. Zitat von Jan Zeinstra <jan@delectat.nl>:
Dear Eugen,
This works: so in conclusion, the assumption is that removing the config with: ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" And disabling the fs: ceph fs snapshot mirror disable proofs
yields: ceph fs snapshot mirror peer_list prodfs Error EINVAL: filesystem prodfs is not mirrored
Reenabling prodfs mirrorring: ceph fs snapshot mirror enable prodfs ceph fs snapshot mirror peer_list prodfs
yields: {}
So: very grateful for your help, and happily proceeding with re-enabling the mirroring solution. Again: Thank you,
Jan Zeinstra
Op ma 14 apr 2025 om 10:29 schreef Eugen Block <eblock@nde.ag>:
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work:
site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored
I assume that this should suffice to start from scratch, but maybe you can confirm.
Zitat von Eugen Block <eblock@nde.ag>:
Please don't drop the list from your responses.
Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
Hi,
I would expect that you have a similar config-key entry:
ceph config-key ls |grep "peer/cephfs" "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56",
Maybe removing that peer would already suffice?
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software.
The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See
http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication ',
but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ?
Best regards, Jan Zeinstra _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, I've had difficulties; my setup was about 6 to 9 months old. My difficulties arose from mixing old idea's and documentation with newer/ more accurate info. As i recall, in the end, i've used IBM documentation in conjunction with Ceph proper documentation. My current (home brew) documentation clearly (helps for me) states what happens on the source cluster, and what happens on the target cluster. Also, of course, the order in which things are to be done, and how to verify the success of every step. Also not entirely clear is why and how onboarding of the target cluster works; A magic token is produced, and somehow the source cluster knows how to contact the remote host: issues with DNS and reachability lurk in the dark. At least, that is what bothered me when setting things up. So; when i had things figured out, it all seemed like a straightforward process, but it suffices to say that the first time is saw bits moving it felt like a hard-fought victory. Ever since then i was worried/ afraid for when something would happen (like outage/ non availabilty of the target system) because it was unclear how to fix issues. Exactly that has happened, and with help of you i'm now looking at a chance to reinstate mirroring on the source system; but troubleshooting/ resetting a mirroring solution is also not exceedingly clearly documented. Hope this helps in forming ideas on how to improve documentation, because [expletive]: what a wonderful solution Ceph is! Not yet disssapointed in the resilience, reliability and versatility! Regards, Jan Zeinstra Op ma 14 apr 2025 om 12:11 schreef Eugen Block <eblock@nde.ag>:
Glad it works for you. May I ask, did you have difficulties setting up mirroring? I tried half a year ago and failed. Then I tried again last week, I succeeded, but I faced several issues along the way. I would like to improve the docs (I'm in contact with Zac about that), but if you didn't encounter any issues and everything was clear, maybe I'm the issue here. ;-) Anyway, if you could share your setup experience, I could compare and figure out which part of the docs could use some improvement.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Dear Eugen,
This works: so in conclusion, the assumption is that removing the config with: ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" And disabling the fs: ceph fs snapshot mirror disable proofs
yields: ceph fs snapshot mirror peer_list prodfs Error EINVAL: filesystem prodfs is not mirrored
Reenabling prodfs mirrorring: ceph fs snapshot mirror enable prodfs ceph fs snapshot mirror peer_list prodfs
yields: {}
So: very grateful for your help, and happily proceeding with re-enabling the mirroring solution. Again: Thank you,
Jan Zeinstra
Op ma 14 apr 2025 om 10:29 schreef Eugen Block <eblock@nde.ag>:
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work:
site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored
I assume that this should suffice to start from scratch, but maybe you can confirm.
Zitat von Eugen Block <eblock@nde.ag>:
Please don't drop the list from your responses.
Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
Hi,
I would expect that you have a similar config-key entry:
ceph config-key ls |grep "peer/cephfs"
"cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56",
Maybe removing that peer would already suffice?
Zitat von Jan Zeinstra <jan@delectat.nl>:
> Hi, > This is my first post to the forum and I don't know if it's
> but I'd like to express my gratitude to all people working hard on ceph > because I think it's a fantastic piece of software. > > The problem I'm having is caused by me; we had a well working ceph fs > mirror solution; let's call it source cluster A, and target cluster B. > Source cluster A is a modest cluster consisting of 6 instances, 3 OSD > instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) > and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is > a single node system having 3 OSD daemons and 3 HDD's. Both clusters run > ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both > systems are installed using cephadm. > I have destroyed cluster B, and have built it from the ground up (I made a > mistake in PG sizing in the original cluster) > Now i find i cannot create/ reinstate the mirroring between 2 ceph fs > filesystems, and i suspect there is a peer left behind in the filesystem of > the source, pointing to the now non-existent target cluster. > When i do 'ceph fs snapshot mirror peer_list prodfs', i get: > '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": > "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' > When i try to delete it: 'ceph fs snapshot mirror peer_remove
> f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to > remove peeraccess denied: does your client key have mgr caps? See > http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication ', > but the logging of the daemon points to the more likely reason of failure: > ---- > Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp > for d0ea284a-8a16-11ee-9232-5934f0f00ec2. > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 > (ceph:ceph) > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process > cephfs-mirror, pid 2 > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty > --pid-file > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register > cephfs-mirror.22849497 metadata > {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 > (e7ad5345525c7a> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0 ] > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 > 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server > allowed_methods [2] but i only support [2,1] > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 > 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: > (13) Permission denied > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: > error connecting to bk-site: (13) Permission denied > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 > 7f57d81e0640 -1 > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cl> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (13) Permission denied > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated > by pthread_kill(), raise(), abort(), alarm()> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) > UID: 0 > ---- > Is there any chance I can get the mirroring daemon to forget about
appropriate, prodfs the
> cluster I lost ? > > Best regards, Jan Zeinstra > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your details, I appreciate it. Is your cephfs-mirror daemon containerized? Because that part didn't work for me, the daemon doesn't know how to contact the primary cluster without a mapping of the remote-site.conf and keyring into the container (I will use extra-container-args for that, similar to rbd-mirror). Also, adding the peer via cluster spec didn't work, I also had to create a token and import it. So there are a bunch of issues (I think I have written them all down somewhere) which are not only a documentation issue, but probably bugs. In the end, it's possible to set it up, but not as easy as it could (or even should?) be. Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi Eugen,
I've had difficulties; my setup was about 6 to 9 months old. My difficulties arose from mixing old idea's and documentation with newer/ more accurate info.
As i recall, in the end, i've used IBM documentation in conjunction with Ceph proper documentation.
My current (home brew) documentation clearly (helps for me) states what happens on the source cluster, and what happens on the target cluster. Also, of course, the order in which things are to be done, and how to verify the success of every step.
Also not entirely clear is why and how onboarding of the target cluster works; A magic token is produced, and somehow the source cluster knows how to contact the remote host: issues with DNS and reachability lurk in the dark. At least, that is what bothered me when setting things up.
So; when i had things figured out, it all seemed like a straightforward process, but it suffices to say that the first time is saw bits moving it felt like a hard-fought victory.
Ever since then i was worried/ afraid for when something would happen (like outage/ non availabilty of the target system) because it was unclear how to fix issues. Exactly that has happened, and with help of you i'm now looking at a chance to reinstate mirroring on the source system; but troubleshooting/ resetting a mirroring solution is also not exceedingly clearly documented.
Hope this helps in forming ideas on how to improve documentation, because [expletive]: what a wonderful solution Ceph is! Not yet disssapointed in the resilience, reliability and versatility!
Regards,
Jan Zeinstra
Op ma 14 apr 2025 om 12:11 schreef Eugen Block <eblock@nde.ag>:
Glad it works for you. May I ask, did you have difficulties setting up mirroring? I tried half a year ago and failed. Then I tried again last week, I succeeded, but I faced several issues along the way. I would like to improve the docs (I'm in contact with Zac about that), but if you didn't encounter any issues and everything was clear, maybe I'm the issue here. ;-) Anyway, if you could share your setup experience, I could compare and figure out which part of the docs could use some improvement.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Dear Eugen,
This works: so in conclusion, the assumption is that removing the config with: ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" And disabling the fs: ceph fs snapshot mirror disable proofs
yields: ceph fs snapshot mirror peer_list prodfs Error EINVAL: filesystem prodfs is not mirrored
Reenabling prodfs mirrorring: ceph fs snapshot mirror enable prodfs ceph fs snapshot mirror peer_list prodfs
yields: {}
So: very grateful for your help, and happily proceeding with re-enabling the mirroring solution. Again: Thank you,
Jan Zeinstra
Op ma 14 apr 2025 om 10:29 schreef Eugen Block <eblock@nde.ag>:
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work:
site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored
I assume that this should suffice to start from scratch, but maybe you can confirm.
Zitat von Eugen Block <eblock@nde.ag>:
Please don't drop the list from your responses.
Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Thanks for the response, I got: ceph config-key ls |grep "peer/prodfs" "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5",
Since there is no mirroring at all going on, I saw no harm in deleting them both. ceph config-key rm "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" key deleted ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" key deleted
And sure enough: ceph config-key ls |grep "peer/prodfs" resulted in nothing found
However: ceph fs snapshot mirror peer_list proofs gave {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}
Also after redeploying the daemon.
Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing.
Do you have more suggestions, where to look ?
(Mirroring is still stuck in trying to find the now nonexistent remote cluster: cephadm logs --name cephfs-mirror.s1mon.lvlkwp
Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 (ceph:ceph) Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty --pid-file Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register cephfs-mirror.23174196 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4,container_hostna> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS SRV with service name: ceph-mon Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors to contact Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (2) No such file or directory Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 7ff4ed4c1640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (2) No such file or directory )
Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>:
> Hi, > > I would expect that you have a similar config-key entry: > > ceph config-key ls |grep "peer/cephfs" > "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56", > > Maybe removing that peer would already suffice? > > > Zitat von Jan Zeinstra <jan@delectat.nl>: > >> Hi, >> This is my first post to the forum and I don't know if it's appropriate, >> but I'd like to express my gratitude to all people working hard on ceph >> because I think it's a fantastic piece of software. >> >> The problem I'm having is caused by me; we had a well working ceph fs >> mirror solution; let's call it source cluster A, and target cluster B. >> Source cluster A is a modest cluster consisting of 6 instances, 3 OSD >> instances, and 3 mon instances. The OSD instances all have 3 disks > (HDD's) >> and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B > is >> a single node system having 3 OSD daemons and 3 HDD's. Both clusters run >> ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both >> systems are installed using cephadm. >> I have destroyed cluster B, and have built it from the ground up (I made > a >> mistake in PG sizing in the original cluster) >> Now i find i cannot create/ reinstate the mirroring between 2 ceph fs >> filesystems, and i suspect there is a peer left behind in the filesystem > of >> the source, pointing to the now non-existent target cluster. >> When i do 'ceph fs snapshot mirror peer_list prodfs', i get: >> '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": >> "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' >> When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs >> f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to >> remove peeraccess denied: does your client key have mgr caps? See >> http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication > ', >> but the logging of the daemon points to the more likely reason of > failure: >> ---- >> Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp >> for d0ea284a-8a16-11ee-9232-5934f0f00ec2. >> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 >> (ceph:ceph) >> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 >> (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process >> cephfs-mirror, pid 2 >> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty >> --pid-file >> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register >> cephfs-mirror.22849497 metadata >> {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 >> (e7ad5345525c7a> >> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0 ] >> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server >> allowed_methods [2] but i only support [2,1] >> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to > bk-site: >> (13) Permission denied >> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils > connect: >> error connecting to bk-site: (13) Permission denied >> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> 7f57d81e0640 -1 >> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> error connecting to remote cl> >> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> error connecting to remote cluster: (13) Permission denied >> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 >> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> by pthread_kill(), raise(), abort(), alarm()> >> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup > from >> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() > ) >> UID: 0 >> ---- >> Is there any chance I can get the mirroring daemon to forget about the >> cluster I lost ? >> >> Best regards, Jan Zeinstra >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
Dear Eugen, If you mean that my ceph installation is 100% cephadm constructed; yes i think so. ps shows: root 2097665 1 0 08:02 ? 00:00:00 /usr/bin/conmon --api-version 1 -c be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc -u be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc/userdata -p /run/containers/storage/overlay-containers/be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc/userdata/pidfile -n ceph-d0ea284a-8a16-11ee-9232-5934f0f00ec2-*cephfs-mirror-s1mon-lvlkwp* --exit-dir /run/libpod/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc/userdata/oci-log --conmon-pidfile /run/ceph-d0ea284a-8a16-11ee-9232-5934f0f00ec2@ *cephfs-mirror.s1mon.lvlkwp.service-pid* --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc and: root 2097668 2097665 0 08:02 ? 00:00:00 /dev/init -- */usr/bin/cephfs-mirror* -n client.cephfs-mirror.s1mon.lvlkwp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false 167 2097671 2097668 0 08:02 ? 00:02:49 */usr/bin/cephfs-mirror* -n client.cephfs-mirror.s1mon.lvlkwp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false so, /usr/bin/cephfs-mirror seems to be non-containerized,, but i'm not entirely sure. Installation has been like: ceph orch apply cephfs-mirror s1mon ceph fs snapshot mirror enable prodfs and ceph fs snapshot mirror peer_bootstrap import prodfs ...token... So i was not aware about it being containerized or not. Hope this answer means something to you, because i'm not sure whether its containerized or not. Regards, Jan Zeinstra Op ma 14 apr 2025 om 13:25 schreef Eugen Block <eblock@nde.ag>:
Thanks for your details, I appreciate it. Is your cephfs-mirror daemon containerized? Because that part didn't work for me, the daemon doesn't know how to contact the primary cluster without a mapping of the remote-site.conf and keyring into the container (I will use extra-container-args for that, similar to rbd-mirror). Also, adding the peer via cluster spec didn't work, I also had to create a token and import it. So there are a bunch of issues (I think I have written them all down somewhere) which are not only a documentation issue, but probably bugs. In the end, it's possible to set it up, but not as easy as it could (or even should?) be.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi Eugen,
I've had difficulties; my setup was about 6 to 9 months old. My difficulties arose from mixing old idea's and documentation with newer/ more accurate info.
As i recall, in the end, i've used IBM documentation in conjunction with Ceph proper documentation.
My current (home brew) documentation clearly (helps for me) states what happens on the source cluster, and what happens on the target cluster. Also, of course, the order in which things are to be done, and how to verify the success of every step.
Also not entirely clear is why and how onboarding of the target cluster works; A magic token is produced, and somehow the source cluster knows how to contact the remote host: issues with DNS and reachability lurk in the dark. At least, that is what bothered me when setting things up.
So; when i had things figured out, it all seemed like a straightforward process, but it suffices to say that the first time is saw bits moving it felt like a hard-fought victory.
Ever since then i was worried/ afraid for when something would happen (like outage/ non availabilty of the target system) because it was unclear how to fix issues. Exactly that has happened, and with help of you i'm now looking at a chance to reinstate mirroring on the source system; but troubleshooting/ resetting a mirroring solution is also not exceedingly clearly documented.
Hope this helps in forming ideas on how to improve documentation, because [expletive]: what a wonderful solution Ceph is! Not yet disssapointed in the resilience, reliability and versatility!
Regards,
Jan Zeinstra
Op ma 14 apr 2025 om 12:11 schreef Eugen Block <eblock@nde.ag>:
Glad it works for you. May I ask, did you have difficulties setting up mirroring? I tried half a year ago and failed. Then I tried again last week, I succeeded, but I faced several issues along the way. I would like to improve the docs (I'm in contact with Zac about that), but if you didn't encounter any issues and everything was clear, maybe I'm the issue here. ;-) Anyway, if you could share your setup experience, I could compare and figure out which part of the docs could use some improvement.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Dear Eugen,
This works: so in conclusion, the assumption is that removing the config with: ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" And disabling the fs: ceph fs snapshot mirror disable proofs
yields: ceph fs snapshot mirror peer_list prodfs Error EINVAL: filesystem prodfs is not mirrored
Reenabling prodfs mirrorring: ceph fs snapshot mirror enable prodfs ceph fs snapshot mirror peer_list prodfs
yields: {}
So: very grateful for your help, and happily proceeding with re-enabling the mirroring solution. Again: Thank you,
Jan Zeinstra
Op ma 14 apr 2025 om 10:29 schreef Eugen Block <eblock@nde.ag>:
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work:
site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored
I assume that this should suffice to start from scratch, but maybe you can confirm.
Zitat von Eugen Block <eblock@nde.ag>:
Please don't drop the list from your responses.
Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that.
Zitat von Jan Zeinstra <jan@delectat.nl>:
> Thanks for the response, > I got: > ceph config-key ls |grep "peer/prodfs" > "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", > "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5", > > Since there is no mirroring at all going on, I saw no harm in deleting them > both. > ceph config-key rm > "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" > key deleted > ceph config-key rm > "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" > key deleted > > And sure enough: > ceph config-key ls |grep "peer/prodfs" > resulted in nothing found > > However: > ceph fs snapshot mirror peer_list proofs > gave > {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": > "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}} > > Also after redeploying the daemon. > > Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing. > > Do you have more suggestions, where to look ? > > (Mirroring is still stuck in trying to find the now nonexistent remote > cluster: > cephadm logs --name cephfs-mirror.s1mon.lvlkwp > > Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp > for d0ea284a-8a16-11ee-9232-5934f0f00ec2. > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 > (ceph:ceph) > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process > cephfs-mirror, pid 2 > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty > --pid-file > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register > cephfs-mirror.23174196 metadata > {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef > (stable),ceph_version_short=18.2.4,container_hostna> > Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils > connect: error connecting to bk-site: (2) No such file or directory > Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (2) No such file or directory > Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS > SRV with service name: ceph-mon > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors > to contact > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: > (2) No such file or directory > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (2) No such file or directory > ) > > Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>: > >> Hi, >> >> I would expect that you have a similar config-key entry: >> >> ceph config-key ls |grep "peer/cephfs" >> "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56", >> >> Maybe removing that peer would already suffice? >> >> >> Zitat von Jan Zeinstra <jan@delectat.nl>: >> >>> Hi, >>> This is my first post to the forum and I don't know if it's appropriate, >>> but I'd like to express my gratitude to all people working hard on ceph >>> because I think it's a fantastic piece of software. >>> >>> The problem I'm having is caused by me; we had a well working ceph fs >>> mirror solution; let's call it source cluster A, and target cluster B. >>> Source cluster A is a modest cluster consisting of 6 instances, 3 OSD >>> instances, and 3 mon instances. The OSD instances all have 3 disks >> (HDD's) >>> and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B >> is >>> a single node system having 3 OSD daemons and 3 HDD's. Both clusters run >>> ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both >>> systems are installed using cephadm. >>> I have destroyed cluster B, and have built it from the ground up (I made >> a >>> mistake in PG sizing in the original cluster) >>> Now i find i cannot create/ reinstate the mirroring between 2 ceph fs >>> filesystems, and i suspect there is a peer left behind in the filesystem >> of >>> the source, pointing to the now non-existent target cluster. >>> When i do 'ceph fs snapshot mirror peer_list prodfs', i get: >>> '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": >>> "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' >>> When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs >>> f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to >>> remove peeraccess denied: does your client key have mgr caps? See >>>
>> ', >>> but the logging of the daemon points to the more likely reason of >> failure: >>> ---- >>> Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp >>> for d0ea284a-8a16-11ee-9232-5934f0f00ec2. >>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 >>> (ceph:ceph) >>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 >>> (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),
>>> cephfs-mirror, pid 2 >>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty >>> --pid-file >>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register >>> cephfs-mirror.22849497 metadata >>> {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 >>> (e7ad5345525c7a> >>> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >>> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >>> remote monitor host=[v2: 172.17.16.12:3300/0,v1:172.17.16.12:6789/0 ] >>> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >>> 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server >>> allowed_methods [2] but i only support [2,1] >>> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >>> 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to >> bk-site: >>> (13) Permission denied >>> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils >> connect: >>> error connecting to bk-site: (13) Permission denied >>> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >>> 7f57d81e0640 -1 >>> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >>> error connecting to remote cl> >>> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >>> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >>> error connecting to remote cluster: (13) Permission denied >>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 >>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>> by pthread_kill(), raise(), abort(), alarm()> >>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >>> UID: 0 >>> ---- >>> Is there any chance I can get the mirroring daemon to forget about
http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication process the
>>> cluster I lost ? >>> >>> Best regards, Jan Zeinstra >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>
Thanks.
ceph orch apply cephfs-mirror s1mon
This means that it's containerized. But for me the mirror doesn't do anything if I have it containerized. But those two lab clusters I'm testing are both on Quincy, maybe I should upgrade to a supported release first... Anyway, thanks again for the details, I'll try to come up with a plan. Zitat von Jan Zeinstra <jan@delectat.nl>:
Dear Eugen,
If you mean that my ceph installation is 100% cephadm constructed; yes i think so. ps shows: root 2097665 1 0 08:02 ? 00:00:00 /usr/bin/conmon --api-version 1 -c be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc -u be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc/userdata -p /run/containers/storage/overlay-containers/be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc/userdata/pidfile -n ceph-d0ea284a-8a16-11ee-9232-5934f0f00ec2-*cephfs-mirror-s1mon-lvlkwp* --exit-dir /run/libpod/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc/userdata/oci-log --conmon-pidfile /run/ceph-d0ea284a-8a16-11ee-9232-5934f0f00ec2@ *cephfs-mirror.s1mon.lvlkwp.service-pid* --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg be2a42bc79e2811d80de4b0f21e9a00786730f64bf829978bc2c00ee26cbf3dc
and: root 2097668 2097665 0 08:02 ? 00:00:00 /dev/init -- */usr/bin/cephfs-mirror* -n client.cephfs-mirror.s1mon.lvlkwp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false 167 2097671 2097668 0 08:02 ? 00:02:49 */usr/bin/cephfs-mirror* -n client.cephfs-mirror.s1mon.lvlkwp -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false
so, /usr/bin/cephfs-mirror seems to be non-containerized,, but i'm not entirely sure.
Installation has been like: ceph orch apply cephfs-mirror s1mon ceph fs snapshot mirror enable prodfs and ceph fs snapshot mirror peer_bootstrap import prodfs ...token...
So i was not aware about it being containerized or not.
Hope this answer means something to you, because i'm not sure whether its containerized or not.
Regards, Jan Zeinstra
Op ma 14 apr 2025 om 13:25 schreef Eugen Block <eblock@nde.ag>:
Thanks for your details, I appreciate it. Is your cephfs-mirror daemon containerized? Because that part didn't work for me, the daemon doesn't know how to contact the primary cluster without a mapping of the remote-site.conf and keyring into the container (I will use extra-container-args for that, similar to rbd-mirror). Also, adding the peer via cluster spec didn't work, I also had to create a token and import it. So there are a bunch of issues (I think I have written them all down somewhere) which are not only a documentation issue, but probably bugs. In the end, it's possible to set it up, but not as easy as it could (or even should?) be.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Hi Eugen,
I've had difficulties; my setup was about 6 to 9 months old. My difficulties arose from mixing old idea's and documentation with newer/ more accurate info.
As i recall, in the end, i've used IBM documentation in conjunction with Ceph proper documentation.
My current (home brew) documentation clearly (helps for me) states what happens on the source cluster, and what happens on the target cluster. Also, of course, the order in which things are to be done, and how to verify the success of every step.
Also not entirely clear is why and how onboarding of the target cluster works; A magic token is produced, and somehow the source cluster knows how to contact the remote host: issues with DNS and reachability lurk in the dark. At least, that is what bothered me when setting things up.
So; when i had things figured out, it all seemed like a straightforward process, but it suffices to say that the first time is saw bits moving it felt like a hard-fought victory.
Ever since then i was worried/ afraid for when something would happen (like outage/ non availabilty of the target system) because it was unclear how to fix issues. Exactly that has happened, and with help of you i'm now looking at a chance to reinstate mirroring on the source system; but troubleshooting/ resetting a mirroring solution is also not exceedingly clearly documented.
Hope this helps in forming ideas on how to improve documentation, because [expletive]: what a wonderful solution Ceph is! Not yet disssapointed in the resilience, reliability and versatility!
Regards,
Jan Zeinstra
Op ma 14 apr 2025 om 12:11 schreef Eugen Block <eblock@nde.ag>:
Glad it works for you. May I ask, did you have difficulties setting up mirroring? I tried half a year ago and failed. Then I tried again last week, I succeeded, but I faced several issues along the way. I would like to improve the docs (I'm in contact with Zac about that), but if you didn't encounter any issues and everything was clear, maybe I'm the issue here. ;-) Anyway, if you could share your setup experience, I could compare and figure out which part of the docs could use some improvement.
Zitat von Jan Zeinstra <jan@delectat.nl>:
Dear Eugen,
This works: so in conclusion, the assumption is that removing the config with: ceph config-key rm "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" And disabling the fs: ceph fs snapshot mirror disable proofs
yields: ceph fs snapshot mirror peer_list prodfs Error EINVAL: filesystem prodfs is not mirrored
Reenabling prodfs mirrorring: ceph fs snapshot mirror enable prodfs ceph fs snapshot mirror peer_list prodfs
yields: {}
So: very grateful for your help, and happily proceeding with re-enabling the mirroring solution. Again: Thank you,
Jan Zeinstra
Op ma 14 apr 2025 om 10:29 schreef Eugen Block <eblock@nde.ag>:
I stopped my peer cluster to simulate it's gone, removing the peer didn't work (although with a different message than yours), but disabling snapshot mirroring did work:
site-a:~ # ceph fs snapshot mirror disable cephfs {} site-a:~ # ceph fs snapshot mirror peer_list cephfs Error EINVAL: filesystem cephfs is not mirrored
I assume that this should suffice to start from scratch, but maybe you can confirm.
Zitat von Eugen Block <eblock@nde.ag>:
> Please don't drop the list from your responses. > > Have you tried disabling mirroring on prodfs after removing the > keys? I haven't had too many chances too play around yet, no prod > cluster using that. > > Zitat von Jan Zeinstra <jan@delectat.nl>: > >> Thanks for the response, >> I got: >> ceph config-key ls |grep "peer/prodfs" >> "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", >> "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5", >> >> Since there is no mirroring at all going on, I saw no harm in deleting them >> both. >> ceph config-key rm >> "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" >> key deleted >> ceph config-key rm >> "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" >> key deleted >> >> And sure enough: >> ceph config-key ls |grep "peer/prodfs" >> resulted in nothing found >> >> However: >> ceph fs snapshot mirror peer_list proofs >> gave >> {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": >> "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}} >> >> Also after redeploying the daemon. >> >> Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing. >> >> Do you have more suggestions, where to look ? >> >> (Mirroring is still stuck in trying to find the now nonexistent remote >> cluster: >> cephadm logs --name cephfs-mirror.s1mon.lvlkwp >> >> Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp >> for d0ea284a-8a16-11ee-9232-5934f0f00ec2. >> Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 >> (ceph:ceph) >> Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 >> (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process >> cephfs-mirror, pid 2 >> Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty >> --pid-file >> Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register >> cephfs-mirror.23174196 metadata >> {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 >> (e7ad5345525c7aa95470c26863873b581076945d) reef >> (stable),ceph_version_short=18.2.4,container_hostna> >> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils >> connect: error connecting to bk-site: (2) No such file or directory >> Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: >> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> error connecting to remote cluster: (2) No such file or directory >> Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS >> SRV with service name: ceph-mon >> Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 >> 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp >> Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 >> 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors >> to contact >> Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 >> 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: >> (2) No such file or directory >> Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 >> 7ff4ed4c1640 -1 >> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> error connecting to remote cluster: (2) No such file or directory >> ) >> >> Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock@nde.ag>: >> >>> Hi, >>> >>> I would expect that you have a similar config-key entry: >>> >>> ceph config-key ls |grep "peer/cephfs" >>> "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56", >>> >>> Maybe removing that peer would already suffice? >>> >>> >>> Zitat von Jan Zeinstra <jan@delectat.nl>: >>> >>>> Hi, >>>> This is my first post to the forum and I don't know if it's appropriate, >>>> but I'd like to express my gratitude to all people working hard on ceph >>>> because I think it's a fantastic piece of software. >>>> >>>> The problem I'm having is caused by me; we had a well working ceph fs >>>> mirror solution; let's call it source cluster A, and target cluster B. >>>> Source cluster A is a modest cluster consisting of 6 instances, 3 OSD >>>> instances, and 3 mon instances. The OSD instances all have 3 disks >>> (HDD's) >>>> and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B >>> is >>>> a single node system having 3 OSD daemons and 3 HDD's. Both clusters run >>>> ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both >>>> systems are installed using cephadm. >>>> I have destroyed cluster B, and have built it from the ground up (I made >>> a >>>> mistake in PG sizing in the original cluster) >>>> Now i find i cannot create/ reinstate the mirroring between 2 ceph fs >>>> filesystems, and i suspect there is a peer left behind in the filesystem >>> of >>>> the source, pointing to the now non-existent target cluster. >>>> When i do 'ceph fs snapshot mirror peer_list prodfs', i get: >>>> '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": >>>> "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' >>>> When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs >>>> f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to >>>> remove peeraccess denied: does your client key have mgr caps? See >>>>
>>> ', >>>> but the logging of the daemon points to the more likely reason of >>> failure: >>>> ---- >>>> Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp >>>> for d0ea284a-8a16-11ee-9232-5934f0f00ec2. >>>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 >>>> (ceph:ceph) >>>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 >>>> (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),
>>>> cephfs-mirror, pid 2 >>>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty >>>> --pid-file >>>> Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register >>>> cephfs-mirror.22849497 metadata >>>> {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 >>>> (e7ad5345525c7a> >>>> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >>>> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >>>> remote monitor host=[v2: 172.17.16.12:3300/0,v1:172.17.16.12:6789/0 ] >>>> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >>>> 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server >>>> allowed_methods [2] but i only support [2,1] >>>> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >>>> 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to >>> bk-site: >>>> (13) Permission denied >>>> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils >>> connect: >>>> error connecting to bk-site: (13) Permission denied >>>> Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >>>> 7f57d81e0640 -1 >>>> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >>>> error connecting to remote cl> >>>> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >>>> cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >>>> error connecting to remote cluster: (13) Permission denied >>>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 >>>> 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >>>> by pthread_kill(), raise(), abort(), alarm()> >>>> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >>> from >>>> Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >>> ) >>>> UID: 0 >>>> ---- >>>> Is there any chance I can get the mirroring daemon to forget about
http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication process the
>>>> cluster I lost ? >>>> >>>> Best regards, Jan Zeinstra >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>
participants (2)
-
Eugen Block
-
Jan Zeinstra