ceph-users
Threads by month
- ----- 2026 -----
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- 42 participants
- 9158 discussions
Hey Mark,
Initially we had it set to use the full 192GB, but after an issue where
starting up the MDS service was going straight to OOM, we brought it down
to 64GB. This has prevented any further crashes. We have a support vendor
that we're working with (OSNexus) at the moment.
On Wed, May 6, 2026 at 10:40 AM Mark Nelson via ceph-users <
ceph-users(a)ceph.io> wrote:
> Hi Max,
>
> Probably the first order of business is to understand the nature of the
> crashes you are seeing. Just as a guess, if you have thousands of jobs
> being run at the same time, you may have a lot of memory pressure on the
> MDSses. If the crashes are just the MDSes going OOM, you might want to
> verify how much of the 192GB of RAM on each node the MDSes are allowed
> to use for cache.
>
> Beyond that, it might take some digging to understand the specifics of
> your issues. There are other institutions that do use CephFS for HPC use
> cases though. I gave a talk at the IO500 BoF at SC24 in collaboration
> with GWDG in Germany regarding their Ceph deployment.
>
> Shameless plug: Clyso does offer academic support packages for upstream
> Ceph if that is of interest to you.
>
> Thanks,
> Mark
>
> On 5/6/26 6:54 AM, Max Breitmeyer via ceph-users wrote:
> > Hello all,
> > At UMBC we were given a grant to run a ceph cluster as our primary
> research
> > storage for an HPC facility. The ceph cluster consists of the following
> > hardware:
> > Mon” nodes x3:
> > • 2x 25Gb interfaces
> > • 192GB RAM
> > • Storage:
> > -- 0 (none) OSDs/drives
> > “HDD” nodes x16:
> > • 2x 25Gb interfaces
> > • 192GB RAM
> > • Storage:
> > -- 12 OSDs w/ 20TB HDDs
> > -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives
> > “NVMe” nodes x3:
> > • 2x 100Gb interfaces
> > • 384GB RAM
> > • Storage:
> > -- 16x OSDs w/ 8TB NVMe drives
> > “MDS” nodes x3:
> > • 2x 100Gb interfaces
> > • 192GB RAM
> > • Storage:
> > -- 8x OSDs w/ 1TB NVMe drives
> >
> >
> > In the six months since moving from a different storage solution
> (isilon),
> > we've had multiple crashes on the system that have completely taken down
> > our system as a result of multiple thousands of jobs being run at the
> same
> > time, causing an overload on the active MDS nodes. Has the use of ceph
> as a
> > primary research storage been used in an HPC with many dozens of
> different
> > workflows and filetypes being used at once? Or is a better to design to
> > come up with a tiered approach in combination with a second storage
> > solution?
> >
> --
> Best Regards,
> Mark Nelson
> Co-Founder and Head of R&D
>
> Clyso Inc.
> p: +49 89 21552391 12 | a: North Vancouver, B.C.
> w: https://clyso.com | e: mark.nelson(a)clyso.com
>
> We are hiring: https://www.clyso.com/jobs/
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
--
V/R,
Maxwell Breitmeyer
UMBC HPCF Specialist
Graduate Student
(443) 835-8250
2
1
Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 08 May '26
by Adiga, Anantha 08 May '26
08 May '26
Thank you Eugen. Truly appreciate your quick response as always.
Regards,
Anantha
-----Original Message-----
From: Eugen Block via ceph-users <ceph-users(a)ceph.io>
Sent: Friday, May 8, 2026 6:15 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
Hi,
I'm afraid I can't really help any further. I don't have any CephFS snapshot mirroring enabled in any of my test clusters at the moment.
The only thing I could quickly check is that on a Squid cluster, I was able to create a snapshot via dashboard of a directory with these
permissions:
/mnt# ll volumes/_nogroup/
total 0
drwxr-sr-x 3 root root 1 May 8 12:43 ./ drwxr-xr-x 3 root root 2 May 8 12:43 ../
d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/
/mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive [
{
"name": "2026-05-08T14:45:13.823+02:00"
}
This works a bit differently in Pacific, I created a subvolume in the same way, changed directory permissions like above, but the Dashboard can't show me the path with this error:
cephfs.OSError: opendir failed: Permission denied [Errno 13]
I can create a snapshot via command line though, so I'm not sure how this error would come into play here. I would expect/hope that upgrading your primary site could resolve this, but of course that's just a guess.
Regards,
Eugen
Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>:
> Hi Eugen,
>
> Regarding this issue, we have 18TB of data successfully mirrored, but
> the process is completely blocked by these permission restrictions.
> CephFS mirroring daemon cannot access directories with restrictive
> permissions, and I'm seeking guidance on the proper solution.
>
> We also find that this issue is not seen on Ceph Reef ( both backup
> source and target )
>
> Environment:
> • Primary Cluster: Ceph Pacific 16.2.5
> • Backup Cluster: Ceph Reef 18.2.7
> • Data Size: ~18TB successfully mirrored before hitting permission issues
>
> Problem Description:
> CephFS mirroring fails when encountering directories with highly
> restrictive permissions (d-----S---). The mirror daemon cannot access
> these directories, causing snapshot synchronization to fail entirely.
>
> Error logs:
> cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat prev
> entry= <path/to/file>: (13) Permission denied
> cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize
> dir_root=/volumes/capi_subvolumegroup/capi_subvolume,
> snapshot=scheduled-2026-04-29-22_00_00
>
> Root Cause Analysis:
> The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph
> arguments, causing it to drop privileges to the ceph user. Even with
> full Ceph capabilities, the MDS enforces POSIX permissions, preventing
> access to directories with no read/execute permissions.
>
> Directory permissions:
> • Many directories with d-----S--- permissions (no access for anyone)
> • These directories contain critical data that must be backed up
> • Cannot modify source file permissions (production system)
>
> Attempted solutions:
>
> [client.cephfs-mirror.a001s017.aoolcx]
> key = <key>
> caps mds = "allow *"
> caps mgr = "allow *"
> caps mon = "allow *"
> caps osd = "allow *"
>
> root@a001s016:~# ceph auth get client.mirror_gfs_src exported keyring
> for client.mirror_gfs_src [client.mirror_gfs_src]
> key = <key>
> caps mds = "allow *"
> caps mgr = "allow *"
> caps mon = "allow *"
> caps osd = "allow *"
> root@a001s016:~#
>
> ceph config set client.cephfs-mirror client_mount_uid 0 ceph config
> set client.cephfs-mirror client_mount_gid 0
>
> ceph config set client.cephfs-mirror setuser 0 ceph config set
> client.cephfs-mirror setgroup 0
>
> Attempted to use extra_entrypoint_args in service spec, but Pacific returns:
> Error EINVAL: ServiceSpec: __init__() got an unexpected keyword
> argument 'extra_entrypoint_args'
>
> Is there a supported way to run cephfs-mirror with root privileges to
> bypass POSIX permission checks?
> Are there service spec parameters in Pacific that allow overriding the
> --setuser/--setgroup arguments?
> Is this a known limitation of CephFS mirroring with restrictive file
> permissions?
> What is the recommended approach for mirroring filesystems containing
> directories that require root access?
>
> Request guidance on:
> • Proper configuration for root-level CephFS mirroring access
> • Whether this is a design limitation or configuration issue
> • Recommended best practices for mirroring restricted filesystems
>
> Regards,
> Anantha
>
> -----Original Message-----
> From: Eugen Block <eblock(a)nde.ag>
> Sent: Thursday, April 23, 2026 1:11 PM
> To: Adiga, Anantha <anantha.adiga(a)intel.com>
> Cc: ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists
> already an active peer" Error (EEXIST) Despite Empty
>
> Awesome, glad I could help!
>
> Zitat von "Adiga, Anantha" <anantha.adiga(a)intel.com>:
>
>> Hi Eugen,
>>
>> Thank you so much. Yes, that fixed the issue:
>> removed the peer uuid from the primary cluster. This removed the
>> attribute from the backup cluster as well. Then import peer bootstrap
>> token worked.
>>
>> Regards,
>> Anantha
>>
>> -----Original Message-----
>> From: Eugen Block via ceph-users <ceph-users(a)ceph.io>
>> Sent: Thursday, April 23, 2026 12:56 AM
>> To: ceph-users(a)ceph.io
>> Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists
>> already an active peer" Error (EEXIST) Despite Empty
>>
>> Hi,
>>
>> a quick look into the code [0] shows (note that I'm not a developer):
>>
>> def set_mirror_info(local_cluster_id, local_fsid, remote_fs):
>> log.info(f'setting {local_cluster_id}::{local_fsid} on remote')
>> try:
>> remote_fs.setxattr('/', 'ceph.mirror.info',
>> f'cluster_id={local_cluster_id}
>> fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE)
>> except cephfs.Error as e:
>> if e.errno == errno.EEXIST:
>>
>>
>> My interpretation is that the setxattr command fails on the remote
>> site. Could this be an auth caps issue? Did you verify the keyrings
>> and their permissions?
>>
>> This is the only lead I got.
>>
>> Regards,
>> Eugen
>>
>> [0]
>> https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20
>> c
>> bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411
>>
>> Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>:
>>
>>> Hi,
>>>
>>> This is issue with CephFS snapshot mirroring setup between two
>>> clusters and would appreciate your assistance.
>>>
>>> Environment:
>>>
>>> * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions)
>>> [cid:image001.png@01DCD199.2933E2E0]
>>> * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions)
>>> [cid:image002.png@01DCD199.2933E2E0]
>>> * Filesystem: cephfs
>>> * Issue: Unable to import peer bootstrap token
>>>
>>> Problem Description:
>>> When attempting to import a peer bootstrap token using:
>>> ceph fs snapshot mirror peer_bootstrap import cephfs <token> I
>>> receive the error:
>>> Error EEXIST: already an active peer mgr.server reply reply (17)
>>> File exists already an active peer However, ceph fs snapshot mirror
>>> peer_list cephfs returns an empty list [], indicating no peers are
>>> configured.
>>>
>>> Steps Performed:
>>> Initial Setup:
>>> 1. Enabled mirroring module: ceph mgr module enable mirroring
>>> 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror
>>> 3. Enabled mirroring on filesystem: ceph fs snapshot mirror
>>> enable cephfs
>>> 4. Created remote user and bootstrap token on backup cluster
>>> 5. Verified mirroring daemons are running: ceph fs snapshot
>>> mirror daemon status
>>> Troubleshooting Attempts:
>>> 1. Reset filesystem mirroring:
>>> ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror
>>> enable cephfs
>>> 2. Module reset:
>>> ceph mgr module disable mirroring
>>> ceph mgr module enable mirroring
>>> 3. Service recreation:
>>> ceph orch rm cephfs-mirror
>>> ceph orch apply cephfs-mirror
>>> 4. Checked for stale configuration:
>>> o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror
>>> o ceph fs snapshot mirror dirmap cephfs returns empty
>>> o ceph fs snapshot mirror status cephfs shows no active mirroring
>>>
>>> Current Status:
>>>
>>> * Mirroring daemons: Running and visible in daemon status
>>> * Peer list: Empty []
>>> * Filesystem mirroring: Enabled
>>> * Error persists: EEXIST despite no visible peers
>>>
>>> Diagnostic Information:
>>>
>>> * MGR logs show: mgr.server reply reply (17) File exists already
>>> an active peer
>>> * No connectivity issues between clusters (ports 6789, 3300,
>>> 6800+ tested)
>>> * Both clusters are healthy with no other issues
>>>
>>> Regards,
>>> Anantha
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
>> email to ceph-users-leave(a)ceph.io
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0
Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Eugen Block 08 May '26
by Eugen Block 08 May '26
08 May '26
Hi,
I'm afraid I can't really help any further. I don't have any CephFS
snapshot mirroring enabled in any of my test clusters at the moment.
The only thing I could quickly check is that on a Squid cluster, I was
able to create a snapshot via dashboard of a directory with these
permissions:
/mnt# ll volumes/_nogroup/
total 0
drwxr-sr-x 3 root root 1 May 8 12:43 ./
drwxr-xr-x 3 root root 2 May 8 12:43 ../
d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/
/mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive
[
{
"name": "2026-05-08T14:45:13.823+02:00"
}
This works a bit differently in Pacific, I created a subvolume in the
same way, changed directory permissions like above, but the Dashboard
can't show me the path with this error:
cephfs.OSError: opendir failed: Permission denied [Errno 13]
I can create a snapshot via command line though, so I'm not sure how
this error would come into play here. I would expect/hope that
upgrading your primary site could resolve this, but of course that's
just a guess.
Regards,
Eugen
Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>:
> Hi Eugen,
>
> Regarding this issue, we have 18TB of data successfully mirrored,
> but the process is completely blocked by these permission
> restrictions. CephFS mirroring daemon cannot access directories with
> restrictive permissions, and I'm seeking guidance on the proper
> solution.
>
> We also find that this issue is not seen on Ceph Reef ( both backup
> source and target )
>
> Environment:
> • Primary Cluster: Ceph Pacific 16.2.5
> • Backup Cluster: Ceph Reef 18.2.7
> • Data Size: ~18TB successfully mirrored before hitting permission issues
>
> Problem Description:
> CephFS mirroring fails when encountering directories with highly
> restrictive permissions (d-----S---). The mirror daemon cannot
> access these directories, causing snapshot synchronization to fail
> entirely.
>
> Error logs:
> cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat
> prev entry= <path/to/file>: (13) Permission denied
> cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize
> dir_root=/volumes/capi_subvolumegroup/capi_subvolume,
> snapshot=scheduled-2026-04-29-22_00_00
>
> Root Cause Analysis:
> The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph
> arguments, causing it to drop privileges to the ceph user. Even with
> full Ceph capabilities, the MDS enforces POSIX permissions,
> preventing access to directories with no read/execute permissions.
>
> Directory permissions:
> • Many directories with d-----S--- permissions (no access for anyone)
> • These directories contain critical data that must be backed up
> • Cannot modify source file permissions (production system)
>
> Attempted solutions:
>
> [client.cephfs-mirror.a001s017.aoolcx]
> key = <key>
> caps mds = "allow *"
> caps mgr = "allow *"
> caps mon = "allow *"
> caps osd = "allow *"
>
> root@a001s016:~# ceph auth get client.mirror_gfs_src
> exported keyring for client.mirror_gfs_src
> [client.mirror_gfs_src]
> key = <key>
> caps mds = "allow *"
> caps mgr = "allow *"
> caps mon = "allow *"
> caps osd = "allow *"
> root@a001s016:~#
>
> ceph config set client.cephfs-mirror client_mount_uid 0
> ceph config set client.cephfs-mirror client_mount_gid 0
>
> ceph config set client.cephfs-mirror setuser 0
> ceph config set client.cephfs-mirror setgroup 0
>
> Attempted to use extra_entrypoint_args in service spec, but Pacific returns:
> Error EINVAL: ServiceSpec: __init__() got an unexpected keyword
> argument 'extra_entrypoint_args'
>
> Is there a supported way to run cephfs-mirror with root privileges
> to bypass POSIX permission checks?
> Are there service spec parameters in Pacific that allow overriding
> the --setuser/--setgroup arguments?
> Is this a known limitation of CephFS mirroring with restrictive file
> permissions?
> What is the recommended approach for mirroring filesystems
> containing directories that require root access?
>
> Request guidance on:
> • Proper configuration for root-level CephFS mirroring access
> • Whether this is a design limitation or configuration issue
> • Recommended best practices for mirroring restricted filesystems
>
> Regards,
> Anantha
>
> -----Original Message-----
> From: Eugen Block <eblock(a)nde.ag>
> Sent: Thursday, April 23, 2026 1:11 PM
> To: Adiga, Anantha <anantha.adiga(a)intel.com>
> Cc: ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File
> exists already an active peer" Error (EEXIST) Despite Empty
>
> Awesome, glad I could help!
>
> Zitat von "Adiga, Anantha" <anantha.adiga(a)intel.com>:
>
>> Hi Eugen,
>>
>> Thank you so much. Yes, that fixed the issue:
>> removed the peer uuid from the primary cluster. This removed the
>> attribute from the backup cluster as well. Then import peer bootstrap
>> token worked.
>>
>> Regards,
>> Anantha
>>
>> -----Original Message-----
>> From: Eugen Block via ceph-users <ceph-users(a)ceph.io>
>> Sent: Thursday, April 23, 2026 12:56 AM
>> To: ceph-users(a)ceph.io
>> Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists
>> already an active peer" Error (EEXIST) Despite Empty
>>
>> Hi,
>>
>> a quick look into the code [0] shows (note that I'm not a developer):
>>
>> def set_mirror_info(local_cluster_id, local_fsid, remote_fs):
>> log.info(f'setting {local_cluster_id}::{local_fsid} on remote')
>> try:
>> remote_fs.setxattr('/', 'ceph.mirror.info',
>> f'cluster_id={local_cluster_id}
>> fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE)
>> except cephfs.Error as e:
>> if e.errno == errno.EEXIST:
>>
>>
>> My interpretation is that the setxattr command fails on the remote
>> site. Could this be an auth caps issue? Did you verify the keyrings
>> and their permissions?
>>
>> This is the only lead I got.
>>
>> Regards,
>> Eugen
>>
>> [0]
>> https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20c
>> bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411
>>
>> Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>:
>>
>>> Hi,
>>>
>>> This is issue with CephFS snapshot mirroring setup between two
>>> clusters and would appreciate your assistance.
>>>
>>> Environment:
>>>
>>> * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions)
>>> [cid:image001.png@01DCD199.2933E2E0]
>>> * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions)
>>> [cid:image002.png@01DCD199.2933E2E0]
>>> * Filesystem: cephfs
>>> * Issue: Unable to import peer bootstrap token
>>>
>>> Problem Description:
>>> When attempting to import a peer bootstrap token using:
>>> ceph fs snapshot mirror peer_bootstrap import cephfs <token> I
>>> receive the error:
>>> Error EEXIST: already an active peer mgr.server reply reply (17) File
>>> exists already an active peer However, ceph fs snapshot mirror
>>> peer_list cephfs returns an empty list [], indicating no peers are
>>> configured.
>>>
>>> Steps Performed:
>>> Initial Setup:
>>> 1. Enabled mirroring module: ceph mgr module enable mirroring
>>> 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror
>>> 3. Enabled mirroring on filesystem: ceph fs snapshot mirror
>>> enable cephfs
>>> 4. Created remote user and bootstrap token on backup cluster
>>> 5. Verified mirroring daemons are running: ceph fs snapshot
>>> mirror daemon status
>>> Troubleshooting Attempts:
>>> 1. Reset filesystem mirroring:
>>> ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable
>>> cephfs
>>> 2. Module reset:
>>> ceph mgr module disable mirroring
>>> ceph mgr module enable mirroring
>>> 3. Service recreation:
>>> ceph orch rm cephfs-mirror
>>> ceph orch apply cephfs-mirror
>>> 4. Checked for stale configuration:
>>> o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror
>>> o ceph fs snapshot mirror dirmap cephfs returns empty
>>> o ceph fs snapshot mirror status cephfs shows no active mirroring
>>>
>>> Current Status:
>>>
>>> * Mirroring daemons: Running and visible in daemon status
>>> * Peer list: Empty []
>>> * Filesystem mirroring: Enabled
>>> * Error persists: EEXIST despite no visible peers
>>>
>>> Diagnostic Information:
>>>
>>> * MGR logs show: mgr.server reply reply (17) File exists already
>>> an active peer
>>> * No connectivity issues between clusters (ports 6789, 3300,
>>> 6800+ tested)
>>> * Both clusters are healthy with no other issues
>>>
>>> Regards,
>>> Anantha
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
>> email to ceph-users-leave(a)ceph.io
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0
08 May '26
Hi,
I think this worked in 19.2.3, but I'm not entirely sure. Have you
tried with latest Tentacle (20.2.1) as well?
Was the bucket also created via dashboard or by the client?
Zitat von farhad kh via ceph-users <ceph-users(a)ceph.io>:
> Ceph Dashboard fails with 500 Internal Server Error when operating on RGW
> buckets that belong to a *tenant*, even though the bucket exists and is
> accessible via radosgw-admin.
>
> RGW returns NoSuchBucket because the Dashboard queries the bucket *without
> the tenant prefix*
> -----------
> 13:02:06.832428866Z"} {"log":"debug 2026-05-04T13:02:06.838+0000
> 7fbc29896640 0 [dashboard ERROR rest_client] RGW REST API failed GET req
> status: 404\n","stream":"stderr","time":"2026-05-04T13:02:06.841073842Z"}
> {"log":"debug 2026-05-04T13:02:06.838+0000 7fbc29896640 0 [dashboard ERROR
> exception] Internal Server
> Error\n","stream":"stderr","time":"2026-05-04T13:02:06.842785189Z"}
> {"log":"Traceback (most recent call
> last):\n","stream":"stderr","time":"2026-05-04T13:02:06.842818184Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/services/exception.py\", line
> 47, in
> dashboard_exception_handler\n","stream":"stderr","time":"2026-05-04T13:02:06.842831624Z"}
> {"log":" return handler(*args,
> **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842841016Z"}
> {"log":" File \"/lib/python3.9/site-packages/cherrypy/_cpdispatch.py\",
> line 54, in
> __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.842851035Z"}
> {"log":" return self.callable(*self.args,
> **self.kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.842983312Z"}
> {"log":" File
> \"/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py\", line
> 264, in inner\n","stream":"stderr","time":"2026-05-04T13:02:06.843004859Z"}
> {"log":" ret = func(*args,
> **kwargs)\n","stream":"stderr","time":"2026-05-04T13:02:06.843015376Z"}
> {"log":" File
> \"/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py\", line
> 193, in
> wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843024549Z"}
> {"log":" return func(*vpath,
> **params)\n","stream":"stderr","time":"2026-05-04T13:02:06.84303376Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line
> 702, in set\n","stream":"stderr","time":"2026-05-04T13:02:06.843050003Z"}
> {"log":" locking = self._get_locking(uid, daemon_name,
> bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843059741Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\", line
> 443, in
> _get_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843069083Z"}
> {"log":" return
> rgw_client.get_bucket_locking(bucket_name)\n","stream":"stderr","time":"2026-05-04T13:02:06.843078375Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 538,
> in
> func_wrapper\n","stream":"stderr","time":"2026-05-04T13:02:06.843087524Z"}
> {"log":" return
> func(\n","stream":"stderr","time":"2026-05-04T13:02:06.843096712Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\",
> line 899, in
> get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843105575Z"}
> {"log":" raise
> e\n","stream":"stderr","time":"2026-05-04T13:02:06.843114638Z"} {"log":"
> File \"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\", line 881, in
> get_bucket_locking\n","stream":"stderr","time":"2026-05-04T13:02:06.843123252Z"}
> {"log":" result = request() # type:
> ignore\n","stream":"stderr","time":"2026-05-04T13:02:06.843132275Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 324,
> in __call__\n","stream":"stderr","time":"2026-05-04T13:02:06.843141087Z"}
> {"log":" resp = self.rest_client.do_request(method, self._gen_path(),
> params,\n","stream":"stderr","time":"2026-05-04T13:02:06.843163504Z"}
> {"log":" File \"/usr/share/ceph/mgr/dashboard/rest_client.py\", line 422,
> in do_request\n","stream":"stderr","time":"2026-05-04T13:02:06.843173809Z"}
> {"log":" raise
> RequestException(\n","stream":"stderr","time":"2026-05-04T13:02:06.843367851Z"}
> {"log":"dashboard.rest_client.RequestException: RGW REST API failed request
> with status code
> 404\n","stream":"stderr","time":"2026-05-04T13:02:06.843382042Z"}
> {"log":"(b'{\"Code\":\"NoSuchBucket\",\"Message\":\"\",\"BucketName\":\"farhad\",\"RequestId\":\"tx000'\n","stream":"stderr","time":"2026-05-04T13:02:06.843391177Z"}
> {"log":"
> b'00ab80c789f95808ff-0069f898ce-110717032-vanak\",\"HostId\":\"110717032-vanak-teh'\n","stream":"stderr","time":"2026-05-04T13:02:06.843400934Z"}
> {"log":"
> b'ran\"}')\n","stream":"stderr","time":"2026-05-04T13:02:06.843410095Z"}
> {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR
> request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.619s] [admin] [513.0B]
> /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.846732997Z"}
> {"log":"debug 2026-05-04T13:02:06.842+0000 7fbc29896640 0 [dashboard ERROR
> request] [b'{\"status\": \"500 Internal Server Error\", \"detail\": \"The
> server encountered an unexpected condition which prevented it from
> fulfilling the request.\", \"request_id\":
> \"a3abea21-d73c-4d26-a8bb-1d99fc53a466\"}
> ']\n","stream":"stderr","time":"2026-05-04T13:02:06.847608634Z"}
> {"log":"debug 2026-05-04T13:02:06.846+0000 7fbc29896640 0 [dashboard INFO
> request] [::ffff:172.20.26.54:49876] [PUT] [500] [0.622s] [admin] [513.0B]
> /api/rgw/bucket/khedrian%2Ffarhad\n","stream":"stderr","time":"2026-05-04T13:02:06.849427429Z"}
> {"log":"debug 2026-05-04T13:02:07.322+0000 7fbc28053640 0 [dashboard INFO
> request] [::ffff:172.20.26.54:64634] [GET] [200] [0.691s] [admin] [295.0B]
> /api/rgw/daemon\n","stream":"stderr","time":"2026-05-04T13:02:07.324402049Z"}
> {"log":"debug 2026-05-04T13:02:07.850+0000 7fbc2504d640 0 [dashboard INFO
> request] [::ffff:172.20.26.54:49870] [GET] [200] [0.004s] [admin] [22.0B]
> /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:07.855794264Z"}
> {"log":"debug 2026-05-04T13:02:08.110+0000 7fbc5c2fb640 0
> log_channel(cluster) log [DBG] : pgmap v611376: 449 pgs: 449 active+clean;
> 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 6.0 KiB/s rd, 0 B/s wr,
> 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:08.113933657Z"}
> {"log":"debug 2026-05-04T13:02:10.111+0000 7fbc5c2fb640 0
> log_channel(cluster) log [DBG] : pgmap v611377: 449 pgs: 449 active+clean;
> 202 MiB data, 10 GiB used, 1.2 TiB / 1.2 TiB avail; 5.7 KiB/s rd, 0 B/s wr,
> 8 op/s\n","stream":"stderr","time":"2026-05-04T13:02:10.115433742Z"}
> {"log":"debug 2026-05-04T13:02:10.211+0000 7fbc2484c640 0 [dashboard INFO
> request] [::ffff:172.20.26.54:49876] [GET] [200] [0.003s] [admin] [22.0B]
> /api/prometheus/notifications\n","stream":"stderr","time":"2026-05-04T13:02:10.2151369Z"}
> ----------------
> root@dev-ceph01:~# ceph -v
> ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle
> (stable)
>
> { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject",
> "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource":
> "arn:aws:s3:::khedrian:farhad/*" } ] }
>
>
> - Bucket exists and is accessible via radosgw-admin
> - Issue only occurs for *tenant buckets*
> - Non‑tenant buckets work correctly
> - Appears to be a *Dashboard RGW tenant parsing bug*
> - Possibly missing tenant propagation when calling get_bucket_locking() and
> similar RGW APIs
> can anyone explain this ? what should i do ?
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
2
Hi,
Going from 16.2.15 in packages to cephadm managed containers and
upgrading to 18.2.8 we find that the InfluxDB MGR module does
not work any more:
# ceph mgr module enable influx
Error ENOENT: module 'influx' reports that it cannot run on the active manager daemon: influxdb python module not found (pass --force to force enablement)
The container image does not contain the necessary Python module.
There is a tracker issue open for 18 months now:
https://tracker.ceph.com/issues/68892
Has the InfluxDB support been deprecated?
Why is it still in the documentation for recent Ceph versions?
Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43
Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG:
HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin
1
0
Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
by Adiga, Anantha 07 May '26
by Adiga, Anantha 07 May '26
07 May '26
Hi Eugen,
Thank you so much. Yes, that fixed the issue:
removed the peer uuid from the primary cluster. This removed the attribute from the backup cluster as well. Then import peer bootstrap token worked.
Regards,
Anantha
-----Original Message-----
From: Eugen Block via ceph-users <ceph-users(a)ceph.io>
Sent: Thursday, April 23, 2026 12:56 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty
Hi,
a quick look into the code [0] shows (note that I'm not a developer):
def set_mirror_info(local_cluster_id, local_fsid, remote_fs):
log.info(f'setting {local_cluster_id}::{local_fsid} on remote')
try:
remote_fs.setxattr('/', 'ceph.mirror.info',
f'cluster_id={local_cluster_id} fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE)
except cephfs.Error as e:
if e.errno == errno.EEXIST:
My interpretation is that the setxattr command fails on the remote site. Could this be an auth caps issue? Did you verify the keyrings and their permissions?
This is the only lead I got.
Regards,
Eugen
[0]
https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20cbd6a/…
Zitat von "Adiga, Anantha via ceph-users" <ceph-users(a)ceph.io>:
> Hi,
>
> This is issue with CephFS snapshot mirroring setup between two
> clusters and would appreciate your assistance.
>
> Environment:
>
> * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions)
> [cid:image001.png@01DCD199.2933E2E0]
> * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions)
> [cid:image002.png@01DCD199.2933E2E0]
> * Filesystem: cephfs
> * Issue: Unable to import peer bootstrap token
>
> Problem Description:
> When attempting to import a peer bootstrap token using:
> ceph fs snapshot mirror peer_bootstrap import cephfs <token> I receive
> the error:
> Error EEXIST: already an active peer mgr.server reply reply (17) File
> exists already an active peer However, ceph fs snapshot mirror
> peer_list cephfs returns an empty list [], indicating no peers are
> configured.
>
> Steps Performed:
> Initial Setup:
> 1. Enabled mirroring module: ceph mgr module enable mirroring
> 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror
> 3. Enabled mirroring on filesystem: ceph fs snapshot mirror
> enable cephfs
> 4. Created remote user and bootstrap token on backup cluster
> 5. Verified mirroring daemons are running: ceph fs snapshot
> mirror daemon status
> Troubleshooting Attempts:
> 1. Reset filesystem mirroring:
> ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable
> cephfs
> 2. Module reset:
> ceph mgr module disable mirroring
> ceph mgr module enable mirroring
> 3. Service recreation:
> ceph orch rm cephfs-mirror
> ceph orch apply cephfs-mirror
> 4. Checked for stale configuration:
> o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror
> o ceph fs snapshot mirror dirmap cephfs returns empty
> o ceph fs snapshot mirror status cephfs shows no active mirroring
>
> Current Status:
>
> * Mirroring daemons: Running and visible in daemon status
> * Peer list: Empty []
> * Filesystem mirroring: Enabled
> * Error persists: EEXIST despite no visible peers
>
> Diagnostic Information:
>
> * MGR logs show: mgr.server reply reply (17) File exists already
> an active peer
> * No connectivity issues between clusters (ports 6789, 3300,
> 6800+ tested)
> * Both clusters are healthy with no other issues
>
> Regards,
> Anantha
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io
2
2
Ceph Steering Committee (CSC) Election Report 2026
The CSC 2026 - Q1 election was conducted according to the guidelines set
forth in section two, part six of the CSC Charter (Adopted Feb 12, 2026),
available at
https://github.com/ceph/ceph/blob/main/doc/technical-charter.rst. With
support and input from current CSC members (Joseph Mundackal, Mark Nelson,
Neha Ojha, and Patrick Donnelly), the next CSC election is scheduled for
October 2026.
This election served as the makeup for the missing 2025 election. This
election cycle ran from March 19 through April 29, with results announced
on May 4. The election was administered by Anthony Middleton, Ceph
Community Manager, using ElectionBuddy as the voting platform, following a
recommendation from the Linux Foundation. This report outlines the
structure of the election, participation levels, outcomes, and key
observations from both stages.
Voting Tool
ElectionBuddy
ElectionBuddy was selected to manage the 2026 CSC election. The platform
supported a multi-stage voting process and introduced capabilities that
aligned more closely with CSC governance needs.
Highlights
-
Supports ranked-choice voting, enabling more flexible and representative
outcomes for leadership roles
-
Streamlined ballot distribution and tracking, with automated email
delivery and vote confirmations
-
Clear audit trails and real-time result visibility for election
administrators
-
Configurable ballots, allowing multiple question types across different
election stages
Election Structure
The 2026 CSC election was conducted in two stages.
Stage 1: Nominations and Governance Updates
Stage 1 included two questions:
1.
Governance Update (Removal Clause)
This question proposed updating the CSC member responsibilities to
include a removal clause. Due to unclear wording, the question caused
confusion among voters and was ultimately discarded. Following a discussion
at the March 16th CSC meeting <https://pad.ceph.com/p/csc-weekly-minutes>,
it was agreed that Anthony Middleton would rewrite the removal section
for clarity before reintroducing it as a GitHub RP.
2.
Co-Chair Nominations
Voting members were asked to nominate up to three CSC members for the
Co-Chair ballot. A list of all active CSC members was provided.
Participation
-
21 ballots were submitted out of 36 eligible voters (58%)
Outcome and Observations
-
Results from the governance question were not counted
-
Nomination results were retained and used to form the final ballot
-
There was confusion about the structure of the election, with some
participants believing Stage 1 was the final vote
-
Several members did not receive ballots due to outdated contact
information on the CSC Governance page
-
Voting was reopened for an additional three days to account for missed
participation
Following Stage 1, outreach was conducted to all individuals nominated. Of
the 11 members nominated, 6 accepted placement on the final ballot.
A new addition to this election cycle was the introduction of candidate
statements. Each candidate was given up to two minutes during a CSC meeting
to speak about their candidacy. This was the first time this format was
used and was received positively by CSC members.
Election Data
ElectionBuddy Repost of Stage 1
<https://docs.google.com/spreadsheets/d/1E1H9GAO9PNGZ-Sl_bOb7PCzEaXhTKPDqZpy…>
Stage 2: Co-Chair Election and Governance Votes
Stage 2 ran for two weeks and included:
-
Two governance questions related to CSC membership removal
-
One ranked-choice vote to elect CSC Co-Chairs
Participation
-
27 ballots submitted out of 36 eligible voters (75%)
-
1 spoiled ballot recorded
Election Data
-
ElectionBuddyReport - Results
<https://docs.google.com/spreadsheets/d/1Q3_jtNfrYcjg77_DGKEYY7-mxXwXTSKvlio…>
-
ElectionBuddyReport -VoteByVote
<https://docs.google.com/spreadsheets/d/1YSw7fSxSO6ucJq58MZAEXhqA7pQE7qJdibY…>
-
ElectionBuddyReport - VoterStatus (voter participation by email)
<https://docs.google.com/spreadsheets/d/1rg0LsAnNgBmIzJtkgSDY7aSA1R0mCSqsMfo…>
Votes for removal of two members
CSC members determined that these members were seen as inactive.
-
Ken Dreyer Removal Vote
-
27 votes in favor
-
0 abstentions
-
Jeff Layton Removal Vote
-
25 votes in favor
-
2 abstentions
Co-Chair Election Results
A ranked voting system was used to elect up to three Co-Chairs. A
governance constraint was applied to ensure that no more than two Co-Chairs
could come from the same organization.
Vote Distribution
-
Patrick Donnelly — 20 votes (27.40%)
-
Neha Ojha — 16 votes (21.92%)
-
Dan van der Ster — 14 votes (19.18%)
-
Joseph Mundackal — 11 votes (15.07%)
-
Ernesto Puerta — 6 votes (8.22%)
-
Gregory Farnum — 6 votes (8.22%)
Outcome
The top three vote-getters were selected as CSC Co-Chairs, subject to the
rule on organizational representation.
Observations
Stage 2 showed clear improvement over Stage 1:
-
Ballot structure and questions were better understood
-
No significant confusion was reported during voting
-
Participation increased from 58% to 75%
However, some challenges remained:
-
A portion of eligible voters did not participate despite multiple
follow-ups via email and Slack
-
Delayed responses from non-participating members slowed the overall
process
Summary
The 2026 CSC election introduced several meaningful improvements, including
ranked-choice voting and candidate statements, both of which contributed to
a more transparent and participatory process. While Stage 1 revealed gaps
in communication and ballot clarity, those issues were addressed in Stage
2, resulting in stronger engagement and cleaner execution.
Future election cycles would benefit from clearer ballot language upfront,
enforced updates to voter contact information, and continued use of
candidate advocacy opportunities.
Best,
-------
Anthony Middleton
Community Manager, Ceph Foundation
amiddleton(a)linuxfoundation.org
LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky
<https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>
1
0
Hi Max,
Probably the first order of business is to understand the nature of the
crashes you are seeing. Just as a guess, if you have thousands of jobs
being run at the same time, you may have a lot of memory pressure on the
MDSses. If the crashes are just the MDSes going OOM, you might want to
verify how much of the 192GB of RAM on each node the MDSes are allowed
to use for cache.
Beyond that, it might take some digging to understand the specifics of
your issues. There are other institutions that do use CephFS for HPC use
cases though. I gave a talk at the IO500 BoF at SC24 in collaboration
with GWDG in Germany regarding their Ceph deployment.
Shameless plug: Clyso does offer academic support packages for upstream
Ceph if that is of interest to you.
Thanks,
Mark
On 5/6/26 6:54 AM, Max Breitmeyer via ceph-users wrote:
> Hello all,
> At UMBC we were given a grant to run a ceph cluster as our primary research
> storage for an HPC facility. The ceph cluster consists of the following
> hardware:
> Mon” nodes x3:
> • 2x 25Gb interfaces
> • 192GB RAM
> • Storage:
> -- 0 (none) OSDs/drives
> “HDD” nodes x16:
> • 2x 25Gb interfaces
> • 192GB RAM
> • Storage:
> -- 12 OSDs w/ 20TB HDDs
> -- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives
> “NVMe” nodes x3:
> • 2x 100Gb interfaces
> • 384GB RAM
> • Storage:
> -- 16x OSDs w/ 8TB NVMe drives
> “MDS” nodes x3:
> • 2x 100Gb interfaces
> • 192GB RAM
> • Storage:
> -- 8x OSDs w/ 1TB NVMe drives
>
>
> In the six months since moving from a different storage solution (isilon),
> we've had multiple crashes on the system that have completely taken down
> our system as a result of multiple thousands of jobs being run at the same
> time, causing an overload on the active MDS nodes. Has the use of ceph as a
> primary research storage been used in an HPC with many dozens of different
> workflows and filetypes being used at once? Or is a better to design to
> come up with a tiered approach in combination with a second storage
> solution?
>
--
Best Regards,
Mark Nelson
Co-Founder and Head of R&D
Clyso Inc.
p: +49 89 21552391 12 | a: North Vancouver, B.C.
w: https://clyso.com | e: mark.nelson(a)clyso.com
We are hiring: https://www.clyso.com/jobs/
1
0
Hello all,
At UMBC we were given a grant to run a ceph cluster as our primary research
storage for an HPC facility. The ceph cluster consists of the following
hardware:
Mon” nodes x3:
• 2x 25Gb interfaces
• 192GB RAM
• Storage:
-- 0 (none) OSDs/drives
“HDD” nodes x16:
• 2x 25Gb interfaces
• 192GB RAM
• Storage:
-- 12 OSDs w/ 20TB HDDs
-- 2 local Write-Access-Logs (journal) w/ 8TB NVMe drives
“NVMe” nodes x3:
• 2x 100Gb interfaces
• 384GB RAM
• Storage:
-- 16x OSDs w/ 8TB NVMe drives
“MDS” nodes x3:
• 2x 100Gb interfaces
• 192GB RAM
• Storage:
-- 8x OSDs w/ 1TB NVMe drives
In the six months since moving from a different storage solution (isilon),
we've had multiple crashes on the system that have completely taken down
our system as a result of multiple thousands of jobs being run at the same
time, causing an overload on the active MDS nodes. Has the use of ceph as a
primary research storage been used in an HPC with many dozens of different
workflows and filetypes being used at once? Or is a better to design to
come up with a tiered approach in combination with a second storage
solution?
--
V/R,
Maxwell Breitmeyer
UMBC HPCF Specialist
Graduate Student
(443) 835-8250
1
0
Hi,
I'm quite sure that we didn't investige MONs at that time, so no, we
haven't tried a leader failover. But it might be possible that they
rebooted nodes, so that would have eventually led to a failover. I
remember that we also did try to rotate keys, that I do have in my
email history, but it didn't have any impact. Waiting also didn't
help, sorry for not having more memories of that incident. :-D
But it's interesting that you could reproduce it with the mon leader,
I wouldn't have expected that.
Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>:
> Hi Eugen,
> Thanks a lot for chiming in — even a partial recollection helps
> narrow the search space.
>
> A few notes from our side, in case it helps differentiate the cases:
>
> - No rbd-mirroring at the customer; single cluster, 39 nodes on Reef 18.2.x.
> - Pattern affects 4 OSDs on 7 out of 39 nodes, persistent over days
> (15M+ log lines in 3 days), no clock skew, quorum stable.
> - We reproduced a very similar log signature on a small test cluster
> (Reef 18.2.7) by isolating a node hosting MON leader + OSDs for >3h,
> matching the original xiaowenhao111 scenario. On our test cluster
> the affected OSDs eventually self-recovered after a couple of
> key-rotation cycles.
> - At the customer, the same signature is NOT clearing on its own.
>
> Two questions on your prior case, if anything comes back to you:
>
> 1. Was zap-and-readd actually required, or was it more of a
> defensive cleanup after another action (e.g. a MON failover, key
> rotation, or simply waiting)? In other words: did you ever observe
> the OSDs unstick on their own, or were they truly stuck?
> 2. Do you remember whether a MON leader failover alone was tried
> before going to OSD rebuild, and whether it had any effect?
>
> The fact that our reproducer self-recovered while the production
> case doesn't suggests there's something cluster-specific keeping the
> rotation desynchronised — possibly stale state on the affected nodes
> rather than a pure timing/quorum effect, which would line up with
> your "remainder from previous tests" intuition.
>
> If nobody points to an existing tracker, we're prepared to open one
> with logs from both the reproduction and the customer environment.
>
> Thanks again,
> Andrea
>
> Il 24/04/26 22:06, Eugen Block via ceph-users ha scritto:
>> Hi,
>>
>> how did you end up in that situation (did something happen in the
>> cluster)? Is it affecting only OSDs or other daemons as well?
>>
>> We did investigate such an issue one and a half years ago for a
>> customer. They added hosts to one of the two clusters, at that time
>> the customer had two sites, with rbd mirroring enabled. And
>> I don't recall the exact details, we did solve it in an online
>> meeting and I don't think anyone wrote a protocoll. And the session
>> definitely wasn't recored. :-) It might have some remainder on the
>> new host from previous tests, or some misconfiguration, I really
>> can't tell anymore. I think in the end we ended up removing the
>> OSDs, zapping them and adding them back. Of course, it could be a
>> different root cause, not sure which role the mon leader can play
>> in this. But I thought I'd share this tiny bit at least. :-)
>>
>> Regards,
>> Eugen
>>
>>
>> Zitat von Andrea Martra via ceph-users <ceph-users(a)ceph.io>:
>>
>>> Hi all,
>>> I'm reaching back to this old thread from December 2023 because
>>> we're investigating a very similar pattern at a customer site
>>> running Ceph Reef 18.2.x. Logs show identical shape:
>>>
>>> AuthRegistry get_handler peer_type 4 method 2 ...
>>> cephx: verify_authorizer decrypted service osd secret_id=3524
>>> auth: could not find secret_id=3524
>>> auth: dump_rotating: (local range ~971-973)
>>> cephx: verify_authorizer could not get service secret ...
>>>
>>> Affecting 4 OSDs on 7 out of 39 nodes, persistent over days
>>> (15M+ log lines in 3 days). No clock skew, quorum stable.
>>>
>>> @xiaowenhao111 or anyone else who hit this: did you ever
>>> find a root cause or a reliable fix?
>>> The original thread
>>> didn't seem to reach a conclusion. Specifically:
>>>
>>> - Did restarting the affected OSDs clear it, or did the
>>> pattern come back?
>>> - Was a MON leader failover enough, or did you need to
>>> rebuild anything?
>>> - Any tracker issue filed for this?
>>>
>>> We're trying to reproduce on a test cluster (Reef 18.2.7)
>>> by isolating a node hosting MON leader + OSDs via network
>>> for 3+ hours, to match the scenario described in the
>>> original thread. Happy to share findings back to the list.
>>>
>>> Thanks,
>>> Andrea
>>>
>>> Il 21/12/23 02:42, xiaowenhao111 ha scritto:
>>>>
>>>>
>>>> Hello ceph-users:
>>>>
>>>> First ,sorry my english ...
>>>>
>>>>
>>>> I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why
>>>>
>>>> test steps:
>>>>
>>>> 1、config all hosts contain chronyd service
>>>>
>>>> 2、cephadm --image quay.io/ceph/ceph:v16.2.10
>>>> <http://quay.io/ceph/ceph:v16.2.10> bootstrap
>>>> --dashboard-password-noupdate --mon-ip 10.40.10.200
>>>> <http://10.40.10.200> --cluster-network=10.40.10.0/24
>>>> <http://10.40.10.0/24> --skip-pull --allow-overwrite
>>>> --skip-monitoring-stack --allow-mismatched-release
>>>>
>>>> 3、add hosts
>>>>
>>>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2
>>>>
>>>> ceph orch host add node2 10.40.10.202 <tel:2104010202> --labels=_admin
>>>>
>>>> ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1
>>>>
>>>> ceph orch host add node-1 10.40.10.202
>>>> <tel:1104010202> --labels=_admin
>>>>
>>>> 4、add osds:
>>>>
>>>> ceph orch daemon add osd node1:/dev/sda
>>>>
>>>> ceph orch daemon add osd node1:/dev/sdb
>>>>
>>>> ceph orch daemon add osd node1:/dev/sdc
>>>>
>>>> ceph orch daemon add osd node1:/dev/sdd
>>>>
>>>> ceph orch daemon add osd node2:/dev/sda
>>>>
>>>> ceph orch daemon add osd node2:/dev/sdb
>>>>
>>>> ceph orch daemon add osd node2:/dev/sdc
>>>>
>>>> ceph orch daemon add osd node2:/dev/sdd
>>>>
>>>> 5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3
>>>>
>>>> 6、disconnect node1 (which is monitor leader) cluster and public network
>>>>
>>>> ifdown enp35s0f0
>>>>
>>>> ifdown enp35s0f1
>>>>
>>>> 7、stay overnight（more than three hours）
>>>>
>>>> 8、connect node1 cluster and public network
>>>>
>>>> ifup enp35s0f0
>>>>
>>>> ifup enp35s0f1
>>>>
>>>>
>>>> then, the osds of node1 call get_auth_session_key from mon.node1
>>>> which is not new leader，get the wrong secret_id=4.
>>>>
>>>> the others report error "could not find secret_id=4" more than 40 minutes
>>>>
>>>>
>>>>
>>>>
>>>> the ceph.log content of node1:
>>>>
>>>>
>>>> 2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800
>>>> mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check
>>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
>>>>
>>>> 2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800
>>>> mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check
>>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
>>>> daemon(s))
>>>>
>>>> 2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800
>>>> mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check
>>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
>>>>
>>>> 2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800
>>>> mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall
>>>> HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm
>>>> daemon(s); 1 pool(s) do not have an application enabled
>>>>
>>>> 2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800
>>>> mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check
>>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
>>>> daemon(s))
>>>>
>>>> 2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800
>>>> mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check
>>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800
>>>> osd.0 (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down,
>>>> but it is still running
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800
>>>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1
>>>> calling monitor election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800
>>>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2
>>>> calling monitor election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800
>>>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor
>>>> election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800
>>>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1
>>>> calling monitor election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800
>>>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is
>>>> new leader, mons node1,node2,node-1 in quorum (ranks 0,1,2)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800
>>>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check
>>>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1)
>>>>
>>>>
>>>> the ceph.log content of node2:
>>>>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800
>>>> mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check
>>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800
>>>> mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check
>>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
>>>> daemon(s))
>>>>
>>>> 2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800
>>>> mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check
>>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800
>>>> mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check
>>>> cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
>>>> daemon(s))
>>>>
>>>> 2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800
>>>> mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check
>>>> failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800
>>>> mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked
>>>> itself dead as of e91
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800
>>>> mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked
>>>> itself dead as of e91
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800
>>>> mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked
>>>> itself dead as of e91
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800
>>>> osd.5 (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down,
>>>> but it is still running
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800
>>>> osd.4 (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down,
>>>> but it is still running
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800
>>>> mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check
>>>> update: Degraded data redundancy: 1/8 objects degraded (12.500
>>>> <tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800
>>>> mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked
>>>> itself dead as of e94
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3
>>>> (osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but
>>>> it is still running
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800
>>>> mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR]
>>>> Failed to load ceph-mgr modules: k8sevents
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800
>>>> osd.0 (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down,
>>>> but it is still running
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800
>>>> mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1
>>>> calling monitor election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800
>>>> mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2
>>>> calling monitor election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800
>>>> mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor
>>>> election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800
>>>> mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1
>>>> calling monitor election
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800
>>>> mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is
>>>> new leader, mons node1,node2,node-1 in quorum (ranks 0,1,2)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800
>>>> mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check
>>>> cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1)
>>>>
>>>>
>>>>
>>>> the osd log of node1:
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.202:6804 <http://10.40.10.202:6804> osd.6 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.202:6812 <http://10.40.10.202:6812> osd.7 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.202:6820 <http://10.40.10.202:6820> osd.8 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.202:6828 <http://10.40.10.202:6828> osd.9 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: reset
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066
>>>> <tel:09562066>bbc000 handle_response ret = 0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: got initial server
>>>> challenge 5fd5d9eca256ae69
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: validate_tickets:
>>>> want=53 need=0 have=53
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have
>>>> 0 need 53
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: build_request
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 20 <tel:864020> cephx client: old ticket len=96
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: get auth session key:
>>>> client_challenge 7ca3f7c3ef30b2bf
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: 0x562066
>>>> <tel:09562066>bbc000 handle_response ret = 0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: get_auth_session_key
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> got 1 keys
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id auth
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> got encrypted ticket
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket.secret_id=2
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> service auth secret_id 2 session_key
>>>> AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== validity=259200.000000
>>>> <tel:259200000000>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket expires=2023-12-24 <tel:20231224>T01:13:44.682217
>>>> <tel:44682217>+0800 renew_after=2023-12-23
>>>> <tel:20231223>T07:13:44.682217 <tel:44682217>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: got connection bl 84
>>>> and extra tickets 550
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: got connection_secret
>>>> 64 bytes
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> got 3 keys
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mon
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket.secret_id=4
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> service mon secret_id 4 session_key
>>>> AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== validity=3600.000000
>>>> <tel:3600000000>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682256
>>>> <tel:44682256>+0800 renew_after=2023-12-21
>>>> <tel:20231221>T01:58:44.682256 <tel:44682256>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id osd
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket.secret_id=4
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> service osd secret_id 4 session_key
>>>> AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== validity=3600.000000
>>>> <tel:3600000000>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682270
>>>> <tel:44682270>+0800 renew_after=2023-12-21
>>>> <tel:20231221>T01:58:44.682270 <tel:44682270>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mgr
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket.secret_id=4
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> service mgr secret_id 4 session_key
>>>> AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== validity=3600.000000
>>>> <tel:3600000000>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
>>>> ticket expires=2023-12-21 <tel:20231221>T02:13:44.682286
>>>> <tel:44682286>+0800 renew_after=2023-12-21
>>>> <tel:20231221>T01:58:44.682286 <tel:44682286>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: got extra
>>>> service_tickets
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have
>>>> 53 need 0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have
>>>> 53 need 0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 20 <tel:864020> cephx client: need_tickets: want=53
>>>> have=53 need=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 -1 <tel:86401> monclient: _check_auth_rotating
>>>> possible clock skew, rotating keys expired way too early (before
>>>> 2023-12-21 <tel:20231221>T00:13:44.682376 <tel:44682376>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
>>>> 7fcd2bed8640 10 <tel:864010> cephx client: build_rotating_request
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.201:6804 <http://10.40.10.201:6804> osd.1 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.201:6812 <http://10.40.10.201:6812> osd.2 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800
>>>> 7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
>>>> 10.40.10.200:6814 <http://10.40.10.200:6814> osd.3 ever on either
>>>> front or back, first ping sent 2023-12-21
>>>> <tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
>>>> deadline 2023-12-21 <tel:20231221>T01:13:43.148457
>>>> <tel:43148457>+0800)
>>>>
>>>>
>>>>
>>>>
>>>> the osd log of node2:
>>>>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
>>>> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have
>>>> 53 need 0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
>>>> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53
>>>> have=53 need=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
>>>> 7fa573a6d640 10 <tel:64010> auth: dump_rotating:
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
>>>> 7fa573a6d640 10 <tel:64010> auth: id 17
>>>> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21
>>>> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800
>>>> 7fa573a6d640 10 <tel:64010> auth: id 18
>>>> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21
>>>> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800
>>>> 7fa573a6d640 10 <tel:64010> auth: id 19
>>>> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21
>>>> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0)
>>>> get_handler peer_type 4 method 2 cluster_methods [2]
>>>> service_methods [2] client_methods [2]
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 10 <tel:58129364010> cephx: verify_authorizer
>>>> decrypted service osd secret_id=4
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 10 <tel:58129364010> auth: dump_rotating:
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 10 <tel:58129364010> auth: id 17
>>>> AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21
>>>> <tel:20231221>T00:34:21.152105 <tel:21152105>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 10 <tel:58129364010> auth: id 18
>>>> AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21
>>>> <tel:20231221>T01:34:25.794799 <tel:25794799>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 10 <tel:58129364010> auth: id 19
>>>> AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21
>>>> <tel:20231221>T02:34:25.794799 <tel:25794799>+0800
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
>>>> 7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could
>>>> not get service secret for service osd secret_id=4
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800
>>>> 7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have
>>>> 53 need 0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800
>>>> 7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53
>>>> have=53 need=0
>>>>
>>>>
>>>>
>>>> the mon log of node1:
>>>>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> start_session
>>>> entity_name=mgr.node1.apderh global_id=14184
>>>> <tel:14184> is_new_global_id=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> start_session server_challenge 524ce5eea3e4dd75
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> handle_request get_auth_session_key for mgr.node1.apderh
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh:
>>>> checking key: req.key=f41952 <tel:41952>c0f02e6c0f
>>>> expected_key=f41952 <tel:41952>c0f02e6c0f
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh:
>>>> checking old_ticket: secret_id=2 len=112,
>>>> old_ticket_may_be_omitted=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh:
>>>> decoded old_ticket: global_id=14184 <tel:14184>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> allowing reclaim of global_id 14184 <tel:14184> (valid ticket
>>>> presented, will encrypt new ticket)
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply
>>>> encoding 1 tickets with secret
>>>> AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg==
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
>>>> auth secret_id 2 ticket_info.ticket.name
>>>> <http://ticket_info.ticket.name>=mgr.node1.apderh
>>>> ticket.global_id 14184 <tel:14184>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
>>>> name=mgr.node1.apderh
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
>>>> of caps=3
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> adding key for service mon
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> adding key for service mds
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
>>>> name=mgr.node1.apderh
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
>>>> of caps=3
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> adding key for service osd
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
>>>> name=mgr.node1.apderh
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
>>>> of caps=3
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
>>>> adding key for service mgr
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
>>>> name=mgr.node1.apderh
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
>>>> of caps=3
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply
>>>> encoding 4 tickets with secret
>>>> AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A==
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
>>>> mon secret_id 4 ticket_info.ticket.name
>>>> <http://ticket_info.ticket.name>=mgr.node1.apderh
>>>> ticket.global_id 14184 <tel:14184>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
>>>> mds secret_id 4 ticket_info.ticket.name
>>>> <http://ticket_info.ticket.name>=mgr.node1.apderh
>>>> ticket.global_id 14184 <tel:14184>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
>>>> osd secret_id 4 ticket_info.ticket.name
>>>> <http://ticket_info.ticket.name>=mgr.node1.apderh
>>>> ticket.global_id 14184 <tel:14184>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
>>>> 7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
>>>> mgr secret_id 4 ticket_info.ticket.name
>>>> <http://ticket_info.ticket.name>=mgr.node1.apderh
>>>> ticket.global_id 14184 <tel:14184>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800
>>>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3
>>>> handle_command mon_command({"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} v 0)
>>>> v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800
>>>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] :
>>>> from='mgr.14184 10.40.10.200:0/2514043697
>>>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh'
>>>> cmd=[{"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]:
>>>> dispatch
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800
>>>> 7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3
>>>> handle_command mon_command({"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} v 0)
>>>> v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800
>>>> 7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] :
>>>> from='mgr.14184 10.40.10.200:0/2514043697
>>>> <http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh'
>>>> cmd=[{"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]:
>>>> dispatch
>>>>
>>>>
>>>>
>>>> the mon log of node2:
>>>>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800
>>>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
>>>> {"time_micros": 1703092390293629, "job": 1145, "event":
>>>> "table_file_deletion", "file_number": 2272}
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800
>>>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
>>>> {"time_micros": 1703092390296385, "job": 1145, "event":
>>>> "table_file_deletion", "file_number": 2270}
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800
>>>> 7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
>>>> {"time_micros": 1703092390304703, "job": 1145, "event":
>>>> "table_file_deletion", "file_number": 2269}
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800
>>>> 7f88b66bb640 4 rocksdb:
>>>> [db_impl/db_impl_compaction_flush.cc:1442
>>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual
>>>> compaction starting
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
>>>> 7f88b66bb640 4 rocksdb:
>>>> [db_impl/db_impl_compaction_flush.cc:1442
>>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual
>>>> compaction starting
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
>>>> 7f88b66bb640 4 rocksdb:
>>>> [db_impl/db_impl_compaction_flush.cc:1442
>>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual
>>>> compaction starting
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
>>>> 7f88b66bb640 4 rocksdb:
>>>> [db_impl/db_impl_compaction_flush.cc:1442
>>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual
>>>> compaction starting
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
>>>> 7f88b66bb640 4 rocksdb:
>>>> [db_impl/db_impl_compaction_flush.cc:1442
>>>> <http://db_impl_compaction_flush.cc:1442>] [default] Manual
>>>> compaction starting
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800
>>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
>>>> mon_command({"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} v 0)
>>>> v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800
>>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
>>>> from='mgr.14252 10.40.10.201:0/357060750
>>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
>>>> cmd=[{"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]:
>>>> dispatch
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800
>>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
>>>> mon_command({"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} v 0)
>>>> v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800
>>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
>>>> from='mgr.14252 10.40.10.201:0/357060750
>>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
>>>> cmd=[{"prefix":"config
>>>> rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]:
>>>> dispatch
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800
>>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
>>>> mon_command({"prefix": "config dump", "format": "json"} v 0) v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800
>>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] :
>>>> from='mgr.14252 10.40.10.201:0/357060750
>>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
>>>> cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800
>>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
>>>> mon_command({"prefix": "config generate-minimal-conf"} v 0) v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800
>>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] :
>>>> from='mgr.14252 10.40.10.201:0/357060750
>>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
>>>> cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800
>>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
>>>> mon_command({"prefix": "auth get", "entity": "client.admin"} v 0)
>>>> v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800
>>>> 7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
>>>> from='mgr.14252 10.40.10.201:0/357060750
>>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
>>>> cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800
>>>> 7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
>>>> mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}]
>>>> v 0) v1
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800
>>>> 7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] :
>>>> from='mgr.14252 10.40.10.201:0/357060750
>>>> <http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800
>>>> 7f88bd6c9640 1 <tel:96401> mon.node2(a)1(leader).osd e91
>>>> _set_new_cache_sizes cache_size:1020054731
>>>> <tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
>>>> 71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800
>>>> 7f88bd6c9640 20 <tel:964020> cephx keyserver:
>>>> prepare_rotating_update before: data.rotating_ver=17
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88beecc640 10 <tel:64010> start_session entity_name=osd.0
>>>> global_id=14205 <tel:14205> is_new_global_id=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88beecc640 10 <tel:64010> cephx server osd.0: start_session
>>>> server_challenge dba982c2bced072d
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.3
>>>> global_id=14379 <tel:14379> is_new_global_id=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88bf6cd640 10 <tel:64010> cephx server osd.3: start_session
>>>> server_challenge e9db92f3f5af188c
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88beecc640 10 <tel:64010> start_session
>>>> entity_name=mgr.node1.apderh global_id=14190
>>>> <tel:14190> is_new_global_id=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88beecc640 10 <tel:64010> cephx server mgr.node1.apderh:
>>>> start_session server_challenge b9809292 <tel:9809292>ea2e62f6
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.5
>>>> global_id=24253 <tel:24253> is_new_global_id=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
>>>> 7f88bf6cd640 10 <tel:64010> cephx server osd.5: start_session
>>>> server_challenge 1be9f89b4bd9cdbc
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800
>>>> 7f88b9ec2640 10 <tel:264010> start_session entity_name=osd.4
>>>> global_id=14406 <tel:14406> is_new_global_id=0
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800
>>>> 7f88b9ec2640 10 <tel:264010> cephx server osd.4: start_session
>>>> server_challenge cca4694d496c39a8
>>>>
>>>> 2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800
>>>> 7f88beecc640 10 <tel:64010> cephx server osd.0: handle_request
>>>> get_auth_session_key for osd.0
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list --ceph-users(a)ceph.io
>>>> To unsubscribe send an email toceph-users-leave(a)ceph.io
>>>
>>> --
>>> --
>>>
>>> Andrea Martra
>>> Senior Manager
>>> +39 393 9048451
>>>
>>> Open Technologies Sas
>>> Via Michele Buniva, 26 - 10064 Pinerolo (TO)
>>> Web:http://www.oscct.it
>>>
>>> ** Open Source Cloud Computing Technologies **
>>>
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> --
> --
>
> Andrea Martra
> Senior Manager
> +39 393 9048451
>
> Open Technologies Sas
> Via Michele Buniva, 26 - 10064 Pinerolo (TO)
> Web: http://www.oscct.it
>
> ** Open Source Cloud Computing Technologies **
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1
0