- ceph-users - lists.ceph.io

CephFS erasure-coded default data pool
by Eugen Block 05 Nov '26

05 Nov '26

Hi, just out of curiosity, has anyone been CephFS that way? I know it's discouraged in the docs (and we haven't tried it otherwise) but since you can still create a CephFS with the --force flag, I was wondering if there's some long-term experience available in the community. Thanks Eugen

6 8

Understanding the cause of (metadata) pool fill-up
by Loïc Tortay 05 Nov '26

05 Nov '26

Hello, We have a cluster used mostly for CephFS, to which we recently added new disk servers (cephadm managed, running v19.2.6). It's a 45% capacity increase, from 52 to 76 OSD nodes. All new nodes have HDD+external WAL/DB on NVMe drives and some dedicated NVMe drives for MD (w/ colocated WAL/DB). The OSDs were added using the method described here: https://docs.clyso.com/docs/kb/rados/adding-hosts-osds/ We also increased the numbe of PGs in the main data (EC) pool (at the same time), before releasing the balancer following https://docs.clyso.com/docs/kb/rados/pg-splitting-with-upmap-remapped/ It went OK for a few days, then we received an alert for the metadata pool being full. Indeed it became full since "something" used all the space on the SSDs that hold the metadata and the "default" pools. These pools are 3-way replicated. The SSDs used for these pools have a colocated WAL/DB. Graphics from our monitoring showing the reduction in MD space available and actual MD usage: https://hidoc.in2p3.fr/s/VsVaULFB9v The reduction of available space (downward slope on the first graph) in the metadata pool started with the rebalance of the data, until zero bytes were left free in the pool. For instance one of the (old) MD OSD: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 16 ssd 0.21829 1.00000 224 GiB 211 GiB 209 GiB 1.4 GiB 1.6 GiB 12 GiB 94.58 1.87 3 up What consummes 200+ GiB on each MD OSD when the total MD pool usage is less than 200 GiB ? Is it the colocated WAL/DB ? There is no "ceph.dir.layout.pool" attribute targeting the "cephfs_ssd" or "cephfs_default" pool (no client has the "p" permission). More details: At some point the smaller MD OSD were really full (> 99% occupation), they all crashed and refuse to start (they are down). Of course the FS is down too. ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 17 ssd 0.21829 1.00000 224 GiB 223 GiB 216 GiB 1.1 GiB 6.4 GiB 424 MiB 99.81 1.97 0 down We managed to add extra OSDs for MD to try to restart, but we are progressing very slowly. "ceph df" (even in detailled mode) does not show what is using that SSD space (this is the state after several days of recovery effort, the FS is not online yet, recovery is still ongoing): [ceph: root@$ADMNODE2 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 18 PiB 8.7 PiB 9.4 PiB 9.4 PiB 51.93 ssd 682 TiB 555 TiB 127 TiB 127 TiB 18.64 TOTAL 19 PiB 9.3 PiB 9.5 PiB 9.5 PiB 50.75 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 2 8 1.0 GiB 265 3.1 GiB 0 37 TiB cephfs_EC_data 3 16384 7.1 PiB 3.15G 8.9 PiB 63.49 4.1 PiB cephfs_metadata 4 1024 193 GiB 91.25M 578 GiB 0.51 37 TiB cephfs_default 5 128 258 MiB 1.34G 773 MiB 0 37 TiB cephfs_ssd 6 128 0 B 0 0 B 0 37 TiB .rgw.root 7 32 1.9 KiB 4 76 KiB 0 1.7 PiB default.rgw.log 8 32 6.8 KiB 210 760 KiB 0 1.7 PiB default.rgw.control 9 32 0 B 8 0 B 0 1.7 PiB default.rgw.meta 10 32 39 KiB 53 1013 KiB 0 1.7 PiB default.rgw.buckets.index 11 32 3.2 GiB 209 9.6 GiB 0 1.7 PiB default.rgw.buckets.data 12 32 5.9 TiB 12.42M 18 TiB 0.34 1.7 PiB default.rgw.buckets.non-ec 13 32 16 MiB 3.20k 48 MiB 0 1.7 PiB cephfs_nvme 14 4096 2.7 TiB 1.84M 3.3 TiB 1.23 214 TiB [ceph: root@$ADMNODE2 /]# => The issue is: what still consumes 120+ TiB on SSDs ? The sum of "USED" for all pools besides "cephfs_EC_data" is way lower than 127 TiB ("EC_data" and "nvme" are EC pools). Before the extension we had about 24 TiB of SSDs available for the "cephfs_metadata", "cephfs_default" and "cephfs_ssd" pools (the latter is no longer used and was replaced with the "cephfs_nvme" pool). We do not use snapshots at all on this cluster. There are no purge queue to empty on the various MDS. The "full-ratio" settings are at the default values. There are no CephFS (non admin) clients with the ability to change the destination pool or create snapshots, and I think "ceph df" would show if the SSD pools were abused by a "clever user". (the "cephfs_default" usage is due to our recovery effort, it was 0 bytes before) Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

2 4

NFS-Export for S3 Bucket with non-default realm
by Robert Sander 05 Nov '26

05 Nov '26

Hi, I created a radosgw realm with a name != "default" in my test cluster. When trying to create an NFS export (via dashboard or the CLI) I get an error that it is not possible because the realm is not default. There is no option to specify the realm when doing this task neither on the dashboard not the CLI. As everything else seems to work with a non-default realm name I wonder if this is a bug in Ceph or if the realm should always be named "default". Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 3

Squid NFS
by Tim Holloway 05 Nov '26

05 Nov '26

I finally migrated from reef to squid. It went well, but I got a whole chumbucket of obsolete encryption messages. A lot of which were for services and servers that don't exist anymore. So I removed the obvious ones and rotated the rest. It's possible that I may have deleted something related to NFS, but I don't think so. Just for context. NFS itself, which has always been a little wonky stopped working. I tried re-defining it, but ran into issues relating to the infamous "cmount_path" and "user_id" upgrade problems, so I corrected as indicated, giving this: [ { "access_type": "RW", "clients": [ { "access_type": "RW", "addresses": [ "10.0.1.0/16", "192.168.1.0/24" ], "squash": "no_root_squash" } ], "cluster_id": "nfs2", "export_id": 1, "fsal": { "fs_name": "ceefs", "cmount_path": "/", "name": "CEPH" }, "path": "/", "protocols": [ 4 ], "security_label": false, "squash": "no_root_squash", "transports": [ "TCP" ], "pseudo": "/ceph" } ] This is the result: ceph nfs export apply nfs -i exports.json [ { "msg": "Failed to apply export: Expecting value: line 1 column 1 (char 0)", "pseudo": "/ceph", "state": "error" } ] The message itself is a blatant lie or at least totally unhelpful. No matter where I put the pseudo definition in the JSON, the same message appears and if I omit it, it complains that "pseudo" is missing. I deleted the cluster, but the export remains. I cannot delete the export itself because without the cmount_path, the deletion fails. Which is absurd. Of all the times you should be able to delete something this is it. I've been able to create a new NFS cluster using more modern techniques, but it won't let me have any useful exports until I can flush out the previous bad one.

1 1

Adding a HDD "Tier" for low-perf storage
by Javier Charne 05 Nov '26

05 Nov '26

Hello! Since NVME drives become quite expensive, we are thinking about adding a "HDD" storage tier, for backups and low-performance storage (RBD only. Not RGW nor CephFS) Currently, we're running Squid (19.2.3) on 15 servers with Intel Xeon 6238 x 112 CPUs, 754 GiB RAM, 4 x 25 Gbps NIC interfaces (yes, they're not enough...). Ubuntu 22.04 Total OSDs: 423 x 3.5 TB ~ (1.9 PB) We want to add 5 servers, each with: - 12 x 20 TB HDD, - 4 NVME 2TB for WAL+DB, - 2 SSD for operative system, - 256 GB RAM, - 24 CPU - 4 x 25 Gbps NIC interfaces. One pool with replica 3 and another with EC 3+2 Please feel free to share any comments or experiences you may have on this topic—they are more than welcome. Thanks in advance! Javier.-

4 5

Upgrading to 20.2.4 crash & ceph-exporter key rotation?
by Robert Sander 04 Nov '26

04 Nov '26

Hi, after upgrading a cephadm managed cluster to 20.2.4 the keys for MON, MGR, OSD and MDS are rotated to the new aes256k format. What has to be done for the ceph-exporter, crash, nfs, rgw, smb and other services managed by cephadm and the bootstrap keys? A "ceph orch redeploy service_type" does not update the keys. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 9

'ceph fs new' with multiple options crashes monitor
by Eugen Block 04 Nov '26

04 Nov '26

Hi, I just created [0] and wanted to warn other users about potentially crashing monitors when creating a CephFS with options as described in the docs [1]: > The set option allows to set multiple options supported by fs set > atomically with the creation of the file system. This command doesn't return until I cancel it: ceph fs new test-cephfs test-cephfs-meta test-cephfs-data set allow_standby_replay true One of the monitors crashes multiple times until I cancel the 'ceph fs new' command. This affects both 20.2.4 and 20.2.5. Regards, Eugen [0] https://tracker.ceph.com/issues/82655 [1] https://docs.ceph.com/en/tentacle/cephfs/createfs/

2 4

Tentacle: Grafana could not load SSL certificate after upgrade to 20.2.5
by Eugen Block 04 Nov '26

04 Nov '26

Hi, I just upgraded one of my test clusters to 20.2.5, the upgrade went well except for Grafana. I have my own certificate and key which worked before the upgrade (from 19.2.7). Now Grafana fails with: > could not load SSL certificate: tls: failed to find any PEM data in > certificate input The certmgr lists the existing cert: host:~ # ceph orch certmgr cert ls grafana_cert scope: host certificates nautilus subject countryName: DE ... commonName: host.domain ... validity remaining_days: 13 But the files in /var/lib/ceph/{FSID}/grafana.host/etc/grafana/certs/ only contain a comment: host:~ # cat /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/grafana.nautilus/etc/grafana/certs/cert_* # generated by cephadm # generated by cephadm These config-keys are stored: "mgr/cephadm/cert_store.cert.grafana_cert", "mgr/cephadm/cert_store.cert.grafana_key", "mgr/cephadm/cert_store.key.grafana_key", "mgr/cephadm/spec.grafana", I know I could just replace the cert and key in that directory and restart grafana. But if certmgr already knows there's a valid cert, why does it (re-)generate an empty cert + key? I'm not sure if I'm missing something or of this is by design. I couldn't find anything in the release notes for 20.2.5 or in the Tentacle docs that would explain this. Thanks! Eugen

2 5

Umbrella release, missmatch?
by Marcus 03 Nov '26

03 Nov '26

Hi all, I have so far been installing ceph manually but as I just got hardware delivered for a new ceph cluster I descided to install umbrella with cephadm. We are using debian so on debian trixie I ran: apt install cephadm cephadm add-repo --release umbrella # Verified that cephadm was in version umbrella cephadm bootstrap --mon-ip ...... Output from bootstrap: This is a develpment version of cephadm. Verifying..... .... ERROR: Container release vampier != cephadm release umbrella; Am I missing something or is there some type of missmatch of versions? Many thanks in advance!! Best regards Marcus

3 2

Umbrella: no OSD stats in dashboard/grafana
by Chris Palmer 03 Nov '26

03 Nov '26

Since upgrading a test cluster and a small production cluster from Tentacle to Umbrella, I have no OSD stats (latencies, distribution of PGs per OSD, onode hit ratios) are all empty. The read/write profile is populated however. Other things such as Pools, Hosts, Block Images, etc are showing correctly. The OSD stats were largely there on Tentacle. I have noticed a couple of things in the logs but I suspect they are irrelevant: * Prometheus: Periodic connection refused to tcp/8765 (service discovery) on each of the non-active-mgr nodes. This wasn't there on Tentacle. It is however connecting ok to the active mgr host. * Node-exporter: Periodic err="failed to retrieve XFS stats: strconv.ParseUint: parsing \"xpc\": invalid syntax". This was there on Tentacle. * Active MGR: /lib/python3.12/site-packages/urllib3/connectionpool.py:1075: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ceph4.xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings. [ceph4 is the host running the monitoring stack. Happens once when I enter any of the performance tabs in the dashboard, including those that appear to be working] Before I start digging into an area I'm not familiar with, does this ring a bell with anyone? Thanks, Chris

1 0