Hey everyone, I've got a difficult problem with my CephFS that I haven't been able to make any headway with. Maybe you guys can help? The problem I’m trying to provision a volume on a CephFS, using a Ceph cluster installed on Kubernetes (K3s) using Rook, but I’m running into the following error (from the Events in |kubectl describe|: |Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m24s default-scheduler Successfully assigned archie/ceph-loader-7989b64fb5-m8ph6 to archie Normal SuccessfulAttachVolume 4m24s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267" Warning FailedMount 3m18s kubelet MountVolume.MountDevice failed for volume "pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph csi-cephfs-node.1@039a3dba-d55c-476f-90f0-8783a18338aa.main-ceph-fs=/volumes/csi/csi-vol-25d616f5-918f-4e15-bfd6-55b866f9aa9f/4bda56a4-5088-451c-90c8-baa83317d5a5 /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/3e10b46e93bcc2c4d3d1b343af01ee628c736ffee7e562e99d478bc397dab10d/globalmount -o mon_addr=10.43.233.111:3300/10.43.237.205:3300/10.43.39.81:3300,secretfile=/tmp/csi/keys/keyfile-2996214224,_netdev] stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized| I’m kind of new to K8s, and /very/ new to Ceph, so I would love some advice on how to go about debugging this mess. General context *Kubernetes distribution*: K3s *Kubernetes version(s)*: v1.33.4+k3s1 (master), v1.32.7+k3s1 (workers) *Ceph*: installed via Rook *Nodes*: 3 *OS*: Linux (Arch on master, NixOS on workers) What I’ve checked/tried *Note*: Since this is a Rook deployment of Ceph (on Kubernetes), all these checks are performed in the Rook Toolbox <https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-toolbox/> container. MDS status / Ceph cluster health Even I know this is the first go-to when your Ceph cluster is giving you issues. I have the Rook toolbox <https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-toolbox/> running on my K8s cluster, so I went into the toolbox pod and ran: |$ ceph status cluster: id: 039a3dba-d55c-476f-90f0-8783a18338aa health: HEALTH_OK| services: mon: 3 daemons, quorum a,c,b (age 7d) mgr: b(active, since 7d), standbys: a mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 7d), 3 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 81 pgs objects: 47 objects, 3.2 MiB usage: 139 MiB used, 502 GiB / 502 GiB avail pgs: 81 active+clean io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr Since the error we started out with |mount error: no mds (Metadata Server) is up|, I checked the |ceph status| output above for the status of the metadata server. As you can see, all the MDS instances are running. CephFS Status |$ ceph fs status main-ceph-fs - 0 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active main-ceph-fs-a Reqs: 0 /s 143 38 37 0 0-s standby-replay main-ceph-fs-b Evts: 0 /s 159 30 29 0 POOL TYPE USED AVAIL main-ceph-fs-metadata metadata 4176k 158G main-ceph-fs-replicated data 0 158G main-ceph-fs-main-ceph-fs-replicated data 0 158G STANDBY MDS main-ceph-fs-d main-ceph-fs-c MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)| Ceph authorizations for MDS Since the other part of the error indicated that I might not be authorized, I wanted to check what the authorizations were: |$ ceph auth ls mds.main-ceph-fs-a # main MDS for my CephFS instance key: <base64 key> caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow * mds.main-ceph-fs-b # standby MDS for my CephFS instance key: <different base64 key> caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow * ... client.csi-cephfs-node.1 # the client mentioned in the error message key: <another base64 key> caps: [mds] allow rw caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rwx tag cephfs metadata=*, allow rw tag cephfs data=* ... # more after this| Note: |main-ceph-fs| is the name I gave my CephFS file system. It looks like this should be okay, but I’m not sure. Definitely open to some more insight here. PersistentVolumeClaim binding I checked to make sure the PersistentVolume was provisioned successfully from the PersistentVolumeClaim, and that it bound appropriately: |$ kubectl get pvc -n archie jellyfin-ceph-pvc NAME STATUS VOLUME CAPACITY jellyfin-ceph-pvc Bound pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267 180Gi | Changing the PVC size to something smaller I tried changing the PVC’s size from 180GB to 1GB, since I thought it might be a free space issue,but the error persisted. Turning off firewalls Turned off all firewall to see if it was that, and still no luck. I’m not quite sure where to go from here. What am I missing? What context should I add? What should I try? What should I check? Thank you so much in advance, Martin