- ceph-users - lists.ceph.io

constant increase in osdmap epoch
by Frank Schilder 15 Apr '25

15 Apr '25

Hi all, we observe a problem that has been reported before, but I can't find a resolution. This is related to an earlier thread "failed to load OSD map for epoch 2898146, got 0 bytes" (https://www.spinics.net/lists/ceph-users/msg84485.html) We run an octopus latest cluster and observe a constant increase in osdmap epoch every few seconds. There is no change in the contents between two successive epochs: # diff map.3075085.txt map.3075086.txt 1c1 < epoch 3075085 --- > epoch 3075086 4c4 < modified 2024-11-18T15:38:45.512100+0100 --- > modified 2024-11-18T15:38:47.858092+0100 This is exactly what others reported too, for example, "steady increasing of osd map epoch since octopus" (https://www.spinics.net/lists/ceph-users/msg69443.html) Its a real problem since it dramatically shortens the time window an OSD can be down before its latest OSD map is purged from the cluster. This, in turn, leads to serious follow-up problems with OSD restart as reported in the thread I'm referring to at the beginning. Related to that I also see the mgrs increasing the pgmap version constantly every 2 seconds. However, I believe this is intentional. I don't see this redundant pgp_num_actual setting by the mgrs reported here: https://tracker.ceph.com/issues/51433 . I can't find a resolution anywhere. Any help would be very much appreciated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 7

Who creates /etc/lograte.d/cephadm in 19.2.2?
by Robert Sander 15 Apr '25

15 Apr '25

Hi, I have a cephadm managed 19.2.2 cluster where logrotate is unable to run because of a configuration conflict. The package ceph-common is installed for all the command line tools and provides /etc/logrotate.d/ceph-common to rotate all logfiles in /var/log. Some process now regularily creates /etc/logrotate.d/cephadm to rotate only /var/log/cephadm.log. And this creates a conflict for logrotate which refuses to run in the end. I tried to remove /etc/logrotate.d/cephadm with rm just to see it reappear after a few minutes. The package cephadm is not installed. I have not seen an entry in the MGR or MON log mentioning the creation of this file. Which process is doing this? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 2

Squid: Ceph mgr shows OSD_Unreachable after setting new host address
by Mevludin Blazevic 14 Apr '25

14 Apr '25

Hi all, after updating host addr via "ceph orch host set-addr x y", the alert OSD_UNREACHABLE appears, althouth the OSDs are accessible. The issue persists after I delete OSDs and add them again after zapping the disc. How to resolve this? Best, Mejdi

1 0

Ceph Day Silicon Valley 2025 - Call for Proposals Now Open!
by Neha Ojha 14 Apr '25

14 Apr '25

Dear Ceph Community, We are very excited to announce that Ceph Day is returning to Silicon Valley after seven years! On March 25, 2025, we will be hosting Ceph Day at IBM Almaden Research Center in San Jose, CA. We are now accepting talk proposals! For more details, visit https://ceph.io/en/community/events/2025/ceph-days-almaden/. We look forward to bringing the Ceph community together once again! Cheers, Neha

2 2

UI for Object Gateway S3 ?
by Anthony Fecarotta 14 Apr '25

14 Apr '25

Hi everyone, Thanks to a member of this wonderful mailing list, I successfully, and easily deployed the RADOS Gateway S3 API on each of the nodes in one of our five node Proxmox clusters. I can see managing this at scale from the terminal being a little tricky, but more so inconvenient. Are there any options (closed or open-source) that offer a way GUI or TUI to manage the S3 Buckets? I was able to find this GitHub Gist ( https://gist.github.com/e02c6081b515881aeb97b26016d0c658.git) that lets Minio act as a GUI for "Ceph backed S3," but it hasn't been updated in nine months and it seems to be intended strictly for Kubernetes. Any of your recommendations are appreciated. I started building a TUI with GO, to solve this problem, but that will take some time to finish 🙃 Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

3 5

Re: Cannot reinstate ceph fs mirror because i destroyed the ceph fs mirror peer/ target server
by Eugen Block 14 Apr '25

14 Apr '25

Please don't drop the list from your responses. Have you tried disabling mirroring on prodfs after removing the keys? I haven't had too many chances too play around yet, no prod cluster using that. Zitat von Jan Zeinstra <jan(a)delectat.nl>: > Thanks for the response, > I got: > ceph config-key ls |grep "peer/prodfs" > "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2", > "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5", > > Since there is no mirroring at all going on, I saw no harm in deleting them > both. > ceph config-key rm > "cephfs/mirror/peer/prodfs/f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5" > key deleted > ceph config-key rm > "cephfs/mirror/peer/prodfs/b308e268-c7f9-4401-a69a-e625955087f2" > key deleted > > And sure enough: > ceph config-key ls |grep "peer/prodfs" > resulted in nothing found > > However: > ceph fs snapshot mirror peer_list proofs > gave > {"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": > "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}} > > Also after redeploying the daemon. > > Searching the config for 'bk-site' and '9afbfe8cc1c5' yielded nothing. > > Do you have more suggestions, where to look ? > > (Mirroring is still stuck in trying to find the now nonexistent remote > cluster: > cephadm logs --name cephfs-mirror.s1mon.lvlkwp > > Apr 14 08:02:36 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp > for d0ea284a-8a16-11ee-9232-5934f0f00ec2. > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: set uid:gid to 167:167 > (ceph:ceph) > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process > cephfs-mirror, pid 2 > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: pidfile_write: ignore empty > --pid-file > Apr 14 08:02:36 s1mon cephfs-mirror[2097671]: mgrc service_daemon_register > cephfs-mirror.23174196 metadata > {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 > (e7ad5345525c7aa95470c26863873b581076945d) reef > (stable),ceph_version_short=18.2.4,container_hostna> > Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: cephfs::mirror::Utils > connect: error connecting to bk-site: (2) No such file or directory > Apr 14 08:02:40 s1mon cephfs-mirror[2097671]: > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (2) No such file or directory > Apr 14 08:02:40 s1mon conmon[2097665]: unable to get monitor info from DNS > SRV with service name: ceph-mon > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 failed for service _ceph-mon._tcp > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 monclient: get_monmap_and_config cannot identify monitors > to contact > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: > (2) No such file or directory > Apr 14 08:02:40 s1mon conmon[2097665]: 2025-04-14T06:02:40.831+0000 > 7ff4ed4c1640 -1 > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: > error connecting to remote cluster: (2) No such file or directory > ) > > Op vr 11 apr 2025 om 14:42 schreef Eugen Block <eblock(a)nde.ag>: > >> Hi, >> >> I would expect that you have a similar config-key entry: >> >> ceph config-key ls |grep "peer/cephfs" >> "cephfs/mirror/peer/cephfs/18c02021-8902-4e3f-bc17-eaf48331cc56", >> >> Maybe removing that peer would already suffice? >> >> >> Zitat von Jan Zeinstra <jan(a)delectat.nl>: >> >> > Hi, >> > This is my first post to the forum and I don't know if it's appropriate, >> > but I'd like to express my gratitude to all people working hard on ceph >> > because I think it's a fantastic piece of software. >> > >> > The problem I'm having is caused by me; we had a well working ceph fs >> > mirror solution; let's call it source cluster A, and target cluster B. >> > Source cluster A is a modest cluster consisting of 6 instances, 3 OSD >> > instances, and 3 mon instances. The OSD instances all have 3 disks >> (HDD's) >> > and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B >> is >> > a single node system having 3 OSD daemons and 3 HDD's. Both clusters run >> > ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both >> > systems are installed using cephadm. >> > I have destroyed cluster B, and have built it from the ground up (I made >> a >> > mistake in PG sizing in the original cluster) >> > Now i find i cannot create/ reinstate the mirroring between 2 ceph fs >> > filesystems, and i suspect there is a peer left behind in the filesystem >> of >> > the source, pointing to the now non-existent target cluster. >> > When i do 'ceph fs snapshot mirror peer_list prodfs', i get: >> > '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": >> > "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' >> > When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs >> > f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to >> > remove peeraccess denied: does your client key have mgr caps? See >> > http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication >> ', >> > but the logging of the daemon points to the more likely reason of >> failure: >> > ---- >> > Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp >> > for d0ea284a-8a16-11ee-9232-5934f0f00ec2. >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 >> > (ceph:ceph) >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 >> > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process >> > cephfs-mirror, pid 2 >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty >> > --pid-file >> > Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register >> > cephfs-mirror.22849497 metadata >> > {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 >> > (e7ad5345525c7a> >> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >> > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> > remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] >> > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> > 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server >> > allowed_methods [2] but i only support [2,1] >> > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> > 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to >> bk-site: >> > (13) Permission denied >> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils >> connect: >> > error connecting to bk-site: (13) Permission denied >> > Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 >> > 7f57d81e0640 -1 >> > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> > error connecting to remote cl> >> > Apr 08 12:54:30 s1mon cephfs-mirror[310088]: >> > cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: >> > error connecting to remote cluster: (13) Permission denied >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 >> > 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated >> > by pthread_kill(), raise(), abort(), alarm()> >> > Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup >> from >> > Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() >> ) >> > UID: 0 >> > ---- >> > Is there any chance I can get the mirroring daemon to forget about the >> > cluster I lost ? >> > >> > Best regards, Jan Zeinstra >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

2 7

OSDs ignore memory limit
by Jonas Schwab 14 Apr '25

14 Apr '25

Hello everyone, I recently have many problems with OSDs using much more memory than they are supposed to (> 10GB), leading to the node running out of memory and killing processes. Does someone have ideas why the daemons seem to completely ignore the set memory limits? See e.g. the following: $ ceph orch ps ceph2-03 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.ceph2-03 ceph2-03 running (3h) 1s ago 2y 501M 2048M 19.2.1 f2efb0401a30 d876fc30f741 node-exporter.ceph2-03 ceph2-03 *:9100 running (3h) 1s ago 17M 46.5M - 1.7.0 72c9c2088986 d32ec4d266ea osd.4 ceph2-03 running (26m) 1s ago 2y 10.2G 3310M 19.2.1 f2efb0401a30 b712a86dacb2 osd.11 ceph2-03 running (5m) 1s ago 2y 3458M 3310M 19.2.1 f2efb0401a30 f3d7705325b4 osd.13 ceph2-03 running (3h) 1s ago 6d 2059M 3310M 19.2.1 f2efb0401a30 980ee7e11252 osd.17 ceph2-03 running (114s) 1s ago 2y 3431M 3310M 19.2.1 f2efb0401a30 be7319fda00b osd.23 ceph2-03 running (30m) 1s ago 2y 10.4G 3310M 19.2.1 f2efb0401a30 9cfb86c4b34a osd.29 ceph2-03 running (8m) 1s ago 2y 4923M 3310M 19.2.1 f2efb0401a30 d764930bb557 osd.35 ceph2-03 running (14m) 1s ago 2y 7029M 3310M 19.2.1 f2efb0401a30 6a4113adca65 osd.59 ceph2-03 running (2m) 1s ago 2y 2821M 3310M 19.2.1 f2efb0401a30 8871d6d4f50a osd.61 ceph2-03 running (49s) 1s ago 2y 1090M 3310M 19.2.1 f2efb0401a30 3f7a0ed17ac2 osd.67 ceph2-03 running (7m) 1s ago 2y 4541M 3310M 19.2.1 f2efb0401a30 eea0a6bcefec osd.75 ceph2-03 running (3h) 1s ago 2y 1239M 3310M 19.2.1 f2efb0401a30 5a801902340d Best regards, Jonas -- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat(a)listserv.dfn.de Email: jonas.schwab(a)uni-wuerzburg.de Tel: +49 931 31-84460

7 9

nodes with high density of OSDs
by Alex from North 13 Apr '25

13 Apr '25

Hello everybody! I have a 4 nodes with 112 OSDs each and 18.2.4. OSD consist of db on SSD and data on HDD For some reason, when I reboot node, not all OSDs get up because some VG or LV are not active. To make it alive again I manually do vgchange -ay $VG_NAME or lvchange -ay $LV_NAME. I suspect it is linked to high amount of vg/lv but cannot find an answer. Maybe you can gimme a hint how to struggle it over?

9 22

FS not mount after update to quincy
by Iban Cabrillo 12 Apr '25

12 Apr '25

Hi guys Good morning, Since I performed the update to Quincy, I've noticed a problem that wasn't present with Octopus. Currently, our Ceph cluster exports a filesystem to certain nodes, which we use as a backup repository. The machines that mount this FS are currently running Ubuntu 24 with Ceph Squid as the client version. zeus22:~ # ls -la /cephvmsfs/ total 225986576 drwxrwxrwx 13 root root 17 Apr 4 13:10 . drwxr-xr-x 1 root root 286 Mar 19 13:27 .. -rw-r--r-- 1 root root 124998647808 Apr 4 13:18 arcceal9.img drwxrwxrwx 2 nobody nogroup 2 Jul 12 2018 backup drwxr-xr-x 2 nobody nogroup 1 Oct 18 2017 Default -rw-r--r-- 1 root root 214cat /etc74836480 Mar 26 18:11 ns1.img drwxr-xr-x 2 root root 1 Aug 29 2024 OnlyOffice Before the update, these nodes mounted the FS correctly (even cluster in octopus and clients in squid), and the nodes that haven't been restarted are still accessing it. One of these machines has been reinstalled, and using the same configuration as the nodes that are still mounting this FS, it is unable to mount, giving errors such as: `mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized` 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300:/ /cephvmsfs ceph name=cephvmsfs,secretfile=/etc/ceph/cephvmsfs.secret,noatime,mds_namespace=cephvmsfs,_netdev 0 0 If I change the port to use 6789 (v1) mount error 110 = Connection timed out ceph cluster is healty and msd are up cephmon01:~ # ceph -s cluster: id: 6f5a65a7-yyy-zzzz-xxxx-428608941dd1 health: HEALTH_OK services: mon: 3 daemons, quorum cephmon01,cephmon03,cephmon02 (age 2d) mgr: cephmon02(active, since 7d), standbys: cephmon01, cephmon03 mds: 1/1 daemons up, 1 standby osd: 231 osds: 231 up (since 7d), 231 in (since 9d) rgw: 2 daemons active (2 hosts, 1 zones) Cephmons are available from clients in both ports: zeus:~ # telnet cephmon02 6789 Trying 10.10.3.2... Connected to cephmon02. Escape character is '^]'. ceph v027�� Ҭ zeus01:~ # telnet cephmon02 3300 Trying 10.10.3.2... Connected to cephmon02. Escape character is '^]'. ceph v2 Any advise is welcomed, regards I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

3 4

Cephadm flooding /var/log/ceph/cephadm.log
by Alex 11 Apr '25

11 Apr '25

Hi everyone. My company has paid Ceph support. The tech is saying: "...cephadm package of ceph 5 is having bug. So It generate debug logs even its set for "info" logs..." I have two clusters running, one Ceph 5 and the other Ceph 6 (Quincy). Both of them are sending "DEBUG ---------" messages to cephadm.log. I checked everywhere I can think of and all the logs are set to INFO. Is there really a bug? I can't find any mention of one anywhere on the internet, besides, it seems that if there really is a bug it hasn't been fixed in version 6 either. I tried setting " mgr/cephadm/log_to_cluster_level" and "mgr/cephadm/log_level" to WARN and noting changed. /etc/ceph/ceph.conf has no mention of log level. Any help would be appreciated, Thanks!

5 27