November 2026 - ceph-users

CephFS erasure-coded default data pool
by Eugen Block 05 Nov '26

05 Nov '26

Hi, just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community. Thanks Eugen

6 7

Understanding the cause of (metadata) pool fill-up
by Loïc Tortay 05 Nov '26

05 Nov '26

Hello, We have a cluster used mostly for CephFS, to which we recently added new disk servers (cephadm managed, running v19.2.6). It's a 45% capacity increase, from 52 to 76 OSD nodes. All new nodes have HDD+external WAL/DB on NVMe drives and some dedicated NVMe drives for MD (w/ colocated WAL/DB). The OSDs were added using the method described here: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ We also increased the numbe of PGs in the main data (EC) pool (at the same time), before releasing the balancer following https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ It went OK for a few days, then we received an alert for the metadata pool being full. Indeed it became full since "something" used all the space on the SSDs that hold the metadata and the "default" pools. These pools are 3-way replicated. The SSDs used for these pools have a colocated WAL/DB. Graphics from our monitoring showing the reduction in MD space available and actual MD usage: https://hidoc.in2p3.fr/s/VsVaULFB9v The reduction of available space (downward slope on the first graph) in the metadata pool started with the rebalance of the data, until zero bytes were left free in the pool. For instance one of the (old) MD OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up What consummes 200+ GiB on each MD OSD when the total MD pool usage is less than 200 GiB ? Is it the colocated WAL/DB ? There is no "ceph.dir.layout.pool" attribute targeting the "cephfs_ssd" or "cephfs_default" pool (no client has the "p" permission). More details: At some point the smaller MD OSD were really full (> 99% occupation), they all crashed and refuse to start (they are down). Of course the FS is down too. ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 17 ssd 0.21829 1.00000 224 GiB 223 GiB 216 GiB 1.1 GiB 6.4 GiB 424 MiB 99.81 1.97 0 down We managed to add extra OSDs for MD to try to restart, but we are progressing very slowly. "ceph df" (even in detailled mode) does not show what is using that SSD space (this is the state after several days of recovery effort, the FS is not online yet, recovery is still ongoing): [ceph: root@$ADMNODE2 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 18 PiB 8.7 PiB 9.4 PiB 9.4 PiB 51.93 ssd 682 TiB 555 TiB 127 TiB 127 TiB 18.64 TOTAL 19 PiB 9.3 PiB 9.5 PiB 9.5 PiB 50.75 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 2 8 1.0 GiB 265 3.1 GiB 0 37 TiB cephfs_EC_data 3 16384 7.1 PiB 3.15G 8.9 PiB 63.49 4.1 PiB cephfs_metadata 4 1024 193 GiB 91.25M 578 GiB 0.51 37 TiB cephfs_default 5 128 258 MiB 1.34G 773 MiB 0 37 TiB cephfs_ssd 6 128 0 B 0 0 B 0 37 TiB .rgw.root 7 32 1.9 KiB 4 76 KiB 0 1.7 PiB default.rgw.log 8 32 6.8 KiB 210 760 KiB 0 1.7 PiB default.rgw.control 9 32 0 B 8 0 B 0 1.7 PiB default.rgw.meta 10 32 39 KiB 53 1013 KiB 0 1.7 PiB default.rgw.buckets.index 11 32 3.2 GiB 209 9.6 GiB 0 1.7 PiB default.rgw.buckets.data 12 32 5.9 TiB 12.42M 18 TiB 0.34 1.7 PiB default.rgw.buckets.non-ec 13 32 16 MiB 3.20k 48 MiB 0 1.7 PiB cephfs_nvme 14 4096 2.7 TiB 1.84M 3.3 TiB 1.23 214 TiB [ceph: root@$ADMNODE2 /]# => The issue is: what still consumes 120+ TiB on SSDs ? The sum of "USED" for all pools besides "cephfs_EC_data" is way lower than 127 TiB ("EC_data" and "nvme" are EC pools). Before the extension we had about 24 TiB of SSDs available for the "cephfs_metadata", "cephfs_default" and "cephfs_ssd" pools (the latter is no longer used and was replaced with the "cephfs_nvme" pool). We do not use snapshots at all on this cluster. There are no purge queue to empty on the various MDS. The "full-ratio" settings are at the default values. There are no CephFS (non admin) clients with the ability to change the destination pool or create snapshots, and I think "ceph df" would show if the SSD pools were abused by a "clever user". (the "cephfs_default" usage is due to our recovery effort, it was 0 bytes before) Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

2 4

NFS-Export for S3 Bucket with non-default realm
by Robert Sander 05 Nov '26

05 Nov '26

Hi, I created a radosgw realm with a name != "default" in my test cluster. When trying to create an NFS export (via dashboard or the CLI) I get an error that it is not possible because the realm is not default. There is no option to specify the realm when doing this task neither on the dashboard not the CLI. As everything else seems to work with a non-default realm name I wonder if this is a bug in Ceph or if the realm should always be named "default". Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 3

Adding a HDD "Tier" for low-perf storage
by Javier Charne 05 Nov '26

05 Nov '26

Hello! Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04 Total OSDs: 423 x 3.5 TB ~ (1.9 PB) We want to add 5 servers, each with: - 12 x 20 TB HDD, - 4 NVME 2TB for WAL+DB, - 2 SSD for operative system, - 256 GB RAM, - 24 CPU - 4 x 25 Gbps NIC interfaces. One pool with replica 3 and another with EC 3+2 Please feel free to share any comments or experiences you may have on this topic—they are more than welcome. Thanks in advance! Javier.-

4 5

Upgrading to 20.2.4 crash & ceph-exporter key rotation?
by Robert Sander 04 Nov '26

04 Nov '26

Hi, after upgrading a cephadm managed cluster to 20.2.4 the keys for MON, MGR, OSD and MDS are rotated to the new aes256k format. What has to be done for the ceph-exporter, crash, nfs, rgw, smb and other services managed by cephadm and the bootstrap keys? A "ceph orch redeploy service_type" does not update the keys. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 9

'ceph fs new' with multiple options crashes monitor
by Eugen Block 04 Nov '26

04 Nov '26

Hi, I just created [0] and wanted to warn other users about potentially crashing monitors when creating a CephFS with options as described in the docs [1]: > The set option allows to set multiple options supported by fs set > atomically with the creation of the file system. This command doesn't return until I cancel it: ceph fs new test-cephfs test-cephfs-meta test-cephfs-data set allow_standby_replay true One of the monitors crashes multiple times until I cancel the 'ceph fs new' command. This affects both 20.2.4 and 20.2.5. Regards, Eugen [0] https://tracker.ceph.com/issues/82655 [1] https://docs.ceph.com/en/tentacle/cephfs/createfs/

2 4

Tentacle: Grafana could not load SSL certificate after upgrade to 20.2.5
by Eugen Block 04 Nov '26

04 Nov '26

Hi, I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7). Now Grafana fails with: > could not load SSL certificate: tls: failed to find any PEM data in > certificate input The certmgr lists the existing cert: host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13 But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment: host:~ # cat /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_* # generated by cephadm # generated by cephadm These config-keys are stored: "mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana", I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this. Thanks! Eugen

2 5

Umbrella release, missmatch?
by Marcus 03 Nov '26

03 Nov '26

Hi all, I have so far been installing ceph manually but as I just got hardware delivered for a new ceph cluster I descided to install umbrella with cephadm. We are using debian so on debian trixie I ran: apt install cephadm cephadm add-repo --release umbrella # Verified that cephadm was in version umbrella cephadm bootstrap --mon-ip ...... Output from bootstrap: This is a develpment version of cephadm. Verifying..... .... ERROR: Container release vampier != cephadm release umbrella; Am I missing something or is there some type of missmatch of versions? Many thanks in advance!! Best regards Marcus

3 2

Squid NFS
by Tim Holloway 03 Nov '26

03 Nov '26

I finally migrated from reef to squid. It went well, but I got a whole chumbucket of obsolete encryption messages. A lot of which were for services and servers that don't exist anymore. So I removed the obvious ones and rotated the rest. It's possible that I may have deleted something related to NFS, but I don't think so. Just for context. NFS itself, which has always been a little wonky stopped working. I tried re-defining it, but ran into issues relating to the infamous "cmount_path" and "user_id" upgrade problems, so I corrected as indicated, giving this: [ { "access_type": "RW", "clients": [ { "access_type": "RW", "addresses": [ "10.0.1.0/16", "192.168.1.0/24" ], "squash": "no_root_squash" } ], "cluster_id": "nfs2", "export_id": 1, "fsal": { "fs_name": "ceefs", "cmount_path": "/", "name": "CEPH" }, "path": "/", "protocols": [ 4 ], "security_label": false, "squash": "no_root_squash", "transports": [ "TCP" ], "pseudo": "/ceph" } ] This is the result: ceph nfs export apply nfs -i exports.json [ { "msg": "Failed to apply export: Expecting value: line 1 column 1 (char 0)", "pseudo": "/ceph", "state": "error" } ] The message itself is a blatant lie or at least totally unhelpful. No matter where I put the pseudo definition in the JSON, the same message appears and if I omit it, it complains that "pseudo" is missing. I deleted the cluster, but the export remains. I cannot delete the export itself because without the cmount_path, the deletion fails. Which is absurd. Of all the times you should be able to delete something this is it. I've been able to create a new NFS cluster using more modern techniques, but it won't let me have any useful exports until I can flush out the previous bad one.

1 0

Umbrella: no OSD stats in dashboard/grafana
by Chris Palmer 03 Nov '26

03 Nov '26

Since upgrading a test cluster and a small production cluster from Tentacle to Umbrella, I have no OSD stats (latencies, distribution of PGs per OSD, onode hit ratios) are all empty. The read/write profile is populated however. Other things such as Pools, Hosts, Block Images, etc are showing correctly. The OSD stats were largely there on Tentacle. I have noticed a couple of things in the logs but I suspect they are irrelevant: * Prometheus: Periodic connection refused to tcp/8765 (service discovery) on each of the non-active-mgr nodes. This wasn't there on Tentacle. It is however connecting ok to the active mgr host. * Node-exporter: Periodic err="failed to retrieve XFS stats: strconv.ParseUint: parsing \"xpc\": invalid syntax". This was there on Tentacle. * Active MGR: /lib/python3.12/site-packages/urllib3/connectionpool.py:1075: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ceph4.xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings. [ceph4 is the host running the monitoring stack. Happens once when I enter any of the performance tabs in the dashboard, including those that appear to be working] Before I start digging into an area I'm not familiar with, does this ring a bell with anyone? Thanks, Chris

1 0

Vote for the W Ceph release name!
by Patrick Donnelly 02 Nov '26

02 Nov '26

Please give your +1 here: https://pad.ceph.com/p/w -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

CSC minutes 2026-11-02
by Yaarit Hatuka 02 Nov '26

02 Nov '26

* Working groups and motions * should we adopt a new policy for LLM? * see: https://github.com/ceph/ceph/pull/72565 * moving https://tracker.ceph.com/issues/82128 to a vote. * this is a copy ofhttps://tracker.ceph.com/issues/79745 which did not pass due to lack of votes * other Voting Active motions: * https://tracker.ceph.com/issues/82119 * https://tracker.ceph.com/issues/82118 * https://tracker.ceph.com/issues/81735 - moved to a vote * PR: https://github.com/ceph/ceph/pull/72649 * W Name decision: https://pad.ceph.com/p/w * vote (by adding your +1) * share with the user,dev lists * Vampire dates * branching at the end of January * do not merge big changes right before branching * removing centos9, jammy from main * Squid last release * go through QA tickets * merge window closes November 11 (community calendar is up to date) * Umbrella v21.2.1 aligns with squid * X release names: [https://pad.ceph.com/p/x](https://pad.ceph.com/p/x) * add names * no need to +1 yet * [Joseph M] - (to be discussed next week) Draft PR created [https://github.com/ceph/ceph/pull/73371](https://github.com/ceph/ceph/pull/… (commit<https://github.com/ceph/ceph/pull/73371](https://github.com/ceph/ceph/pull/…> sha 8c048607addd8e23def31bca14b0559222f15d94) for automatic member removal policy * unfortunately on PTO to will miss meeting * Hoping to use this draft to get feedback and make tweaks before putting this to a vote asap to reduce election and governance overheads

1 0

v20.2.5 Tentacle released
by Yuri Weinstein 02 Nov '26

02 Nov '26

This is the fifth minor release in the Tentacle series. We recommend that all users update to this release. Notable Changes Security - CephX: hardened service ticket verification. RGW (RADOS Gateway) - Added AES-256-GCM support for server-side encryption. - Added support for a global CORS rule, and CORS rules are now matched the way Amazon S3 matches them. - Added radosgw-admin bucket suspend and unsuspend commands. - Added --shard-id support to radosgw-admin gc process and gc list. - radosgw-admin bilog trim now requires --yes-i-really-mean-it. - STS Lite is deprecated for Tentacle. - Fixed conditional writes on versioning-suspended buckets. - Fixed SigV4 signature handling so Content-Type is no longer always required in the signature and x-amz-content-sha256 need not be a signed header. - S3 GetUsage now reads the user usage log. - Added Keystone scope information to ops logging. - Fixed several multisite issues: bucket object-lock sync, bucket reshard racing with index updates, and bilog autotrim on archive zones and stale bucket instance metadata. RADOS / OSD / BlueStore - BlueStore: fixed unsharing blobs on remove, stray spanning blobs produced by ExtentMap::reshard, onode cache stalls, and expanded space being added to the allocator after allocmap recovery. - BlueStore: BlueFS spillover cleaner improvements. - Autoscaler: PG autoscaling is now allowed for overlapped CRUSH roots, and PG distribution is fixed. - Scrub now auto-corrects accounting-only stat mismatches. - mClock: bounded the high_priority queue drain to prevent starvation. - pg_upmap_primary mappings are now removed for PG merge targets. - The read balancer no longer aborts on float rounding. - Reverted the msg/async/ProtocolV2 connection-speedup and netsplit grace period change. CephFS - MDS now persists the session auth_name in the ESession journal event. - The client accounts for mixed quotas in statfs. - cephfs-journal-tool can now scan and flush the journal in batches. - Fixed ceph fs perf stats handling of the json-pretty output format. RBD - It's possible to specify the source cluster's mon_host and key for native format migration via the migration spec now. This eliminates the dependency on a <cluster-name>.conf file in a known location, which is rather rigid and also challenging to disseminate in some environments. The key can be embedded in the migration spec or just referenced from there while stored in the MON config-key store. - Fixed a use-after-free in trash purge on image open error. - Fixed list_snaps() handling of non-existent groups in pybind/rbd. cephadm / NVMe-oF - Added a command to send signals to daemons. - Upgrades onto hosts whose CPU cannot run the target release are now blocked. - NVMe-oF gateways are disabled and enabled around host maintenance. - Added CNC and Prometheus frequency fields to the NVMe-oF spec file, --keep-connections to the NVMe-oF CLI, and --force to the NVMe-oF CLI delete kmip server command. - Fixed several NVMe-oF gateway monitor issues, including stretched-cluster relocate and beacon timeouts for gateways in the Created state. Dashboard - Upgraded Angular (19, then 20.3.29). - Backported the notification panel, overview page fixes, and a batch of other dashboard fixes. - OAuth2 SSO configuration fixes. Platform and Packaging - Ubuntu 24.04 is now a supported distro and container host. - Builds now use FUSE 3, and the C standard moved from C99 to C11. - s390x CRC assembly no longer requires an ELF executable stack. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-5-tentacle-released/ Getting Ceph - Git at git://github.com/ceph/ceph.git - Tarball at https://download.ceph.com/tarballs/ceph-20.2.5.tar.gz - Containers at https://quay.io/repository/ceph/ceph - For packages, see https://docs.ceph.com/en/latest/install/get-packages/ - Release git sha1: ed27c686770872022f4c1de04f6b0ac5d5384ded

3 5

Massive mon slow-ops when restarting mon daemons
by Alex Walender 02 Nov '26

02 Nov '26

Hey there! I’m currently in the process of upgrading the production cluster from v18.2.8 -> v19.2.6 via „ceph orch upgrade“. I tested this upgrade in our staging environment and everything went great. On our production environment I encounter some worrisome issues with the mon daemons. The cluster itself: 5x MON Daemons 5x MGR Daemons 1500x OSD - 2 larger Pools with each 16k in PG. … so a somewhat larger cluster. The orchestrator started re-deploying the MGR, which went fine. During the MON redeployment, the cluster went into warning with messages like „34846 slow ops. mon.mon1 has slow ops“. The counter climbed rapidly. Only way to stop this was to restart the MON in question. Sometimes I had to restart the MON multiple times. Afterwards the cluster is healthy again….until I restart any MON. The symptoms return until restarting fixes it again. I tried to dump the mon_ops_in_flight, but the command just hangs (probably because of the large error count). Filtering the journal for the issues: ``` journalctl -g "oldest is" --since "3 hours ago" -u ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c(a)mon.partnach <mailto:ceph-d8a6f365-99de-42f2-b32b-c9ea11df945c@mon.partnach> Sep 14 09:45:05 partnach conmon[1735]: 2026-09-14T07:45:05.331+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18286 slow ops, oldest is log(1000 entries from seq 2 at 2026-08-26T04:51:47.819315+0000) Sep 14 09:45:10 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:10 partnach conmon[1735]: 2026-09-14T07:45:10.827+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18505 slow ops, oldest is log(1000 entries from seq 4003 at 2026-09-11T12:25:52.136679+0000) Sep 14 09:45:15 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:15 partnach conmon[1735]: 2026-09-14T07:45:15.924+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18671 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-07T16:47:58.529125+0000) Sep 14 09:45:21 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:21 partnach conmon[1735]: 2026-09-14T07:45:21.224+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 18814 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-10T20:12:26.565558+0000) Sep 14 09:45:26 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:26 partnach conmon[1735]: 2026-09-14T07:45:26.700+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19042 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-10T14:13:26.625340+0000) Sep 14 09:45:32 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:32 partnach conmon[1735]: 2026-09-14T07:45:32.432+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19280 slow ops, oldest is log(1000 entries from seq 2002 at 2026-09-13T08:23:50.558481+0000) Sep 14 09:45:37 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:37 partnach conmon[1735]: 2026-09-14T07:45:37.568+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19433 slow ops, oldest is log(1000 entries from seq 1002 at 2026-09-11T20:28:14.104877+0000) Sep 14 09:45:43 partnach ceph-mon[1742]: mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) Sep 14 09:45:43 partnach conmon[1735]: 2026-09-14T07:45:43.292+0000 7fe90bfaf640 -1 mon.partnach@0(leader) e78 get_health_metrics reporting 19632 slow ops, oldest is log(1000 entries from seq 2 at 2026-09-10T08:47:13.259138+0000) ``` I’m not quite sure what the lock is here. The mon tries to store some old cluster log entries from various dates? Checked the underlying network fabric, CPU load, clock skew, compacted the rocksdb manually on each mon ect. All is fine. I also set "ceph config set global mon_cluster_log_level info“ **before** the 10th of September. Luckily the upgrade has finished with the MON daemons and is now re-deploying all OSDs. Does anyone know whats going on? Some kind of old hanging Log-Entries? Best Regards, Alex Walender --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

4 4

Umbrella 21.2.0 bad MGR memory leak
by Chris Palmer 02 Nov '26

02 Nov '26

I've just upgraded a small production light-usage cephadm cluster from Tentacle to Umbrella. The active MGR now seems to be growing by about 1GB per hour though. I can't see anything wrong in the mgr logs. The only other sign of trouble is that a lot of metrics are not displaying in Grafana (mainly OSD-related I think - haven't checked it thoroughly yet). I don't know if that is related to the leak. I just had to fail the MGR to get some memory back. It immediately released about 2.5GB, but is still tracking up gently while in standby. Help... (Just adding a large amount of swap...) Thanks, Chris

3 4