Hi, I'm afraid I can't really help any further. I don't have any CephFS snapshot mirroring enabled in any of my test clusters at the moment. The only thing I could quickly check is that on a Squid cluster, I was able to create a snapshot via dashboard of a directory with these permissions: /mnt# ll volumes/_nogroup/ total 0 drwxr-sr-x 3 root root 1 May 8 12:43 ./ drwxr-xr-x 3 root root 2 May 8 12:43 ../ d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/ /mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive [ { "name": "2026-05-08T14:45:13.823+02:00" } This works a bit differently in Pacific, I created a subvolume in the same way, changed directory permissions like above, but the Dashboard can't show me the path with this error: cephfs.OSError: opendir failed: Permission denied [Errno 13] I can create a snapshot via command line though, so I'm not sure how this error would come into play here. I would expect/hope that upgrading your primary site could resolve this, but of course that's just a guess. Regards, Eugen Zitat von "Adiga, Anantha via ceph-users" <ceph-users@ceph.io>:
Hi Eugen,
Regarding this issue, we have 18TB of data successfully mirrored, but the process is completely blocked by these permission restrictions. CephFS mirroring daemon cannot access directories with restrictive permissions, and I'm seeking guidance on the proper solution.
We also find that this issue is not seen on Ceph Reef ( both backup source and target )
Environment: • Primary Cluster: Ceph Pacific 16.2.5 • Backup Cluster: Ceph Reef 18.2.7 • Data Size: ~18TB successfully mirrored before hitting permission issues
Problem Description: CephFS mirroring fails when encountering directories with highly restrictive permissions (d-----S---). The mirror daemon cannot access these directories, causing snapshot synchronization to fail entirely.
Error logs: cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat prev entry= <path/to/file>: (13) Permission denied cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize dir_root=/volumes/capi_subvolumegroup/capi_subvolume, snapshot=scheduled-2026-04-29-22_00_00
Root Cause Analysis: The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph arguments, causing it to drop privileges to the ceph user. Even with full Ceph capabilities, the MDS enforces POSIX permissions, preventing access to directories with no read/execute permissions.
Directory permissions: • Many directories with d-----S--- permissions (no access for anyone) • These directories contain critical data that must be backed up • Cannot modify source file permissions (production system)
Attempted solutions:
[client.cephfs-mirror.a001s017.aoolcx] key = <key> caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"
root@a001s016:~# ceph auth get client.mirror_gfs_src exported keyring for client.mirror_gfs_src [client.mirror_gfs_src] key = <key> caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" root@a001s016:~#
ceph config set client.cephfs-mirror client_mount_uid 0 ceph config set client.cephfs-mirror client_mount_gid 0
ceph config set client.cephfs-mirror setuser 0 ceph config set client.cephfs-mirror setgroup 0
Attempted to use extra_entrypoint_args in service spec, but Pacific returns: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_entrypoint_args'
Is there a supported way to run cephfs-mirror with root privileges to bypass POSIX permission checks? Are there service spec parameters in Pacific that allow overriding the --setuser/--setgroup arguments? Is this a known limitation of CephFS mirroring with restrictive file permissions? What is the recommended approach for mirroring filesystems containing directories that require root access?
Request guidance on: • Proper configuration for root-level CephFS mirroring access • Whether this is a design limitation or configuration issue • Recommended best practices for mirroring restricted filesystems
Regards, Anantha
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Thursday, April 23, 2026 1:11 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
Awesome, glad I could help!
Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:
Hi Eugen,
Thank you so much. Yes, that fixed the issue: removed the peer uuid from the primary cluster. This removed the attribute from the backup cluster as well. Then import peer bootstrap token worked.
Regards, Anantha
-----Original Message----- From: Eugen Block via ceph-users <ceph-users@ceph.io> Sent: Thursday, April 23, 2026 12:56 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
Hi,
a quick look into the code [0] shows (note that I'm not a developer):
def set_mirror_info(local_cluster_id, local_fsid, remote_fs): log.info(f'setting {local_cluster_id}::{local_fsid} on remote') try: remote_fs.setxattr('/', 'ceph.mirror.info', f'cluster_id={local_cluster_id} fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) except cephfs.Error as e: if e.errno == errno.EEXIST:
My interpretation is that the setxattr command fails on the remote site. Could this be an auth caps issue? Did you verify the keyrings and their permissions?
This is the only lead I got.
Regards, Eugen
[0] https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20c bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411
Zitat von "Adiga, Anantha via ceph-users" <ceph-users@ceph.io>:
Hi,
This is issue with CephFS snapshot mirroring setup between two clusters and would appreciate your assistance.
Environment:
* Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) [cid:image001.png@01DCD199.2933E2E0] * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) [cid:image002.png@01DCD199.2933E2E0] * Filesystem: cephfs * Issue: Unable to import peer bootstrap token
Problem Description: When attempting to import a peer bootstrap token using: ceph fs snapshot mirror peer_bootstrap import cephfs <token> I receive the error: Error EEXIST: already an active peer mgr.server reply reply (17) File exists already an active peer However, ceph fs snapshot mirror peer_list cephfs returns an empty list [], indicating no peers are configured.
Steps Performed: Initial Setup: 1. Enabled mirroring module: ceph mgr module enable mirroring 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror 3. Enabled mirroring on filesystem: ceph fs snapshot mirror enable cephfs 4. Created remote user and bootstrap token on backup cluster 5. Verified mirroring daemons are running: ceph fs snapshot mirror daemon status Troubleshooting Attempts: 1. Reset filesystem mirroring: ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable cephfs 2. Module reset: ceph mgr module disable mirroring ceph mgr module enable mirroring 3. Service recreation: ceph orch rm cephfs-mirror ceph orch apply cephfs-mirror 4. Checked for stale configuration: o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror o ceph fs snapshot mirror dirmap cephfs returns empty o ceph fs snapshot mirror status cephfs shows no active mirroring
Current Status:
* Mirroring daemons: Running and visible in daemon status * Peer list: Empty [] * Filesystem mirroring: Enabled * Error persists: EEXIST despite no visible peers
Diagnostic Information:
* MGR logs show: mgr.server reply reply (17) File exists already an active peer * No connectivity issues between clusters (ports 6789, 3300, 6800+ tested) * Both clusters are healthy with no other issues
Regards, Anantha
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io