- ceph-users - lists.ceph.io

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

CSC minutes 2026-03-23
by Yaarit Hatuka 23 Mar '26

23 Mar '26

- (bumped from last week) [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe - why? this can be a solution to some technical decision making deadlocks we have from time to time - came from this discussion: https://ceph-storage.slack.com/archives/C05R975114L/p1772629034963979 - next step: feedback pls -> get the text in shape to vote on -> csc vote -> merge. - Should we consider a mailing list platform migration from postorius to groups.io - +1! - we will get support from LF for this as well - this also reduces the overhead to maitaining the mailing list imo - Ceph Foundation Board and CSC Q1 Meeting on March 30, 10 am PT - Ceph on Windows: Plan of action (Alexandre JARDON) - rocky10 PRs are merged to main - March Ceph Tech Talk speaker needed

1 0

tentacle 20.2.1 QE validation status
by Yuri Weinstein 22 Mar '26

22 Mar '26

Details of this release are summarized here: https://tracker.ceph.com/issues/74838#note-4 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya (still running) nvmeof: Aviv, Vallari upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: Laura upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 2

Re: Merging two clusters
by Nico Schottelius 21 Mar '26

21 Mar '26

Ciao Burkhard, Burkhard Linke via ceph-users <ceph-users(a)ceph.io> writes: > Hi, > > > we used the rbd live migration in the past to split up a cluster. The > processed pools were used for image and volumes in our openstack > cloud. Similar here, just using opennebula. > Live migration mostly works, but if the RBD image is in use, you have > to add some extra effort. Cinder for examples stores the mon > information in its database, and this has to be updated. Similar, plus with opennebula it's embedded in the libvirt XML and not so easy to modify. > I used a > script to migrate a RBD image, pause the instance, adjust the database > entries, and live migrate the instance itself to another host to > update the qemu/libvirt information. If the image is not actively used > migration is trivial; if the image is attached to a stopped virtual > machine migration is also trivial, but database entries have to be > updated. Got it. > The rather old nautilus clusters might be problem, so if you want to > try rbd live migration you need to test this first. Will take that into consideration. The rbd mirror might actually also work for us, because a short downtime can likely be tolerated. > Not sure whether the script still exists, but I can try to dig it out > if necessary. If you still have it, would certainly be interested to take a glimpse at it, at least for inspiration. Sunny greetings from Switzerland, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: Merging two clusters
by Nico Schottelius 21 Mar '26

21 Mar '26

Hey Anthony, Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> writes: > I’ve used rbd-mirror to do this sort of thing in the past. It’s > straightforward with unattached volumes. I had a homebrew harness > that for attached images would handle a reboot of the client with > reattach. That seems to be a way forward from what it seems. There are quite some details to address anyway from what I can see, as clients also need to be reconfigured and partially moved to a different network. Thanks for the pointer! Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: v18.2.8 Reef released
by Eugen Block 21 Mar '26

21 Mar '26

Hi, I just tried the first upgrade in a single node cluster (18.2.7 --> 18.2.8) and would like to share a few notes. The staggered upgrade should handle non-existing daemons a bit smoother. Before upgrading OSDs one has to upgrade the crash service first: ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.8 --daemon-types crash --limit 1 Initiating upgrade to quay.io/ceph/ceph:v18.2.8 And then the upgrade fails: ceph health detail HEALTH_ERR Upgrade: failed due to an unexpected exception [ERR] UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception Unexpected exception occurred during upgrade process: division by zero This is because there are no crash services running. Not sure how common this is, but maybe another check is required if there even are daemons of the specified type. Or at least add some details to the health output. Besides that, the limit parameter doesn't apply to all services. I reported that a few weeks ago (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/6TF5EOBGWC…) Apart from that, the upgrade was successful on my lab cluster, so thanks for the great work! Regards, Eugen Zitat von Yuri Weinstein via ceph-users <ceph-users(a)ceph.io>: > We're happy to announce the 8th, and expected to be last, backport release > in the Reef series. We recommend that all users update to this release. > > We recommend users to update to this release. > For detailed release notes with links & changelog please refer to the > official blog entry at > https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/ > > Known Issues > ------------ > * During QA for v18.2.8, it was found that there was a bug for upgrades from > Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a > deprecated connection feature bit that was adopted to indicate a Reef OSD. > This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually > upgraded to Reef. There are no known issues associated with Pacific and > Reef > OSDs interoperating where Pacific OSDs are "advertising" Reef > compatibility; > however, out of an abundance of caution, we no longer recommend upgrading > from Pacific to Reef directly. > > Security Fixes > -------------- > * CephFS Client: A fix was merged to prohibit unprivileged users from > modifying > the sgid or suid bits on a file. Previously, unprivileged users were > inadvertently permitted to set these bits if they were the sole bits being > modified. > * Mgr Alerts: The SMTP SSL context was enforced in the mgr/alerts module to > resolve a security vulnerability (GHSA-xj9f-7g59-m4jx). > > Notable Changes > --------------- > * RGW (RADOS Gateway): > - Fixed an issue where bucket rm --bypass-gc was mistakenly removing head > objects instead of tail objects, potentially causing data inconsistencies. > - Fixed rgw-restore-bucket-index to handle objects with leading hyphens > and to process versioned buckets correctly. > - Addressed an issue in the msg/async protocol that caused memory locks > and hangs during connection shutdown. > - RGW STS: Made JWKS URL verification configurable for AWS compliance via > the rgw_enable_jwks_url_verification configuration. > > * CephFS / MDS: > - Prevented the MDS from stalling (up to 5 seconds) during rename/stat > workloads by forcing the log to nudge for unstable locks after early > replies. > - Fixed cephfs-journal-tool so it no longer incorrectly resets the > journal trim position during disaster recovery, which was causing stale > journal objects to linger forever in the metadata pool. > - Fixed a bug where ll_walk incorrectly processed absolute paths as > relative paths. > - Prevented the ceph fs volume create command from accidentally deleting > user-created pools if the command aborted during cleanup. > - MDS Batched Operations: Added a new mds_allow_batched_ops configuration > option (default: true) to control whether the MDS can batch lookup or > getattr RPCs. > - CephFS Subvolumes: Added the ceph fs subvolume snapshot getpath command > to allow users to retrieve the absolute path of a snapshot of a subvolume. > > * BlueStore: > - Fixed a bug where the bytes_written_slow performance counter > incorrectly reported 0 when using aio_write. > > Getting Ceph > ------------ > * Git at git://github.com/ceph/ceph.git > * Tarball at https://download.ceph.com/tarballs/ceph-18.2.7.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * Release git sha1: efac5a54607c13fa50d4822e50242b86e6e446df > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: 8443: connect: connection refused
by Eugen Block 21 Mar '26

21 Mar '26

I haven't used mgmt-gateway yet, except for a short test if I could deploy it. But the fact that alertmanager is trying to connect to the wrong dashboard port makes me wonder. Alertmanager should be configured accoording to the dashboard config, for example I bootstrapped a new cluster with the parameter "--ssl-dashboard-port 8080": root@tentacle-ubuntu24:~# ceph mgr services { "dashboard": "https://192.168.124.172:8080/" "prometheus": "http://192.168.124.172:9283/" } After applying alertmanager, the webhook_configs are configured correctly: root@tentacle-ubuntu24:~# grep -A1 webhook_configs /var/lib/ceph/70495e92-2466-11f1-96c3-fa163edcc19e/alertmanager.tentacle-ubuntu24/etc/alertmanager/alertmanager.yml webhook_configs: - name: 'ceph-dashboard' webhook_configs: - url: 'https://tentacle-ubuntu24.domain:8080/api/prometheus_receiver' Did you see anything in the mgr logs related to "reconfigure" the alertmanager? Or did you maybe at some point change the ssl port of the dashboard (manually)? The respective config-key is: config/mgr/mgr/dashboard/ssl_server_port And does the error go away if you run: ceph orch reconfigure alertmanager Or alternatively redeploy it: ceph orch redeploy alertmanager Regards, Eugen Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > I did not think that ceph should be aware of any "external" proxy > being used - it runs on one of ceph nodes - I assumed such a proxy > would only be relevant to users/clients looking at ceph via > web-browser. > > As far a ceph goes I did: > > -> $ ceph config set mgr mgr/dashboard/standby_behaviour "redirect" > # set it back to default > > -> $ ceph config set mgr mgr/dashboard/standby_behaviour "error" > -> $ ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr > true # resolve to hostnames > -> $ ceph config set mgr mgr/dashboard/ssl false > -> $ ceph mgr services > { > "dashboard": "http://10.1.1.63:8080/", > "prometheus": "http://10.1.1.63:9283/" > } > > > Rest is Nginx's, what I'd call a "standard" proxy configuration. > If needed - I should spam it here? > > thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

v18.2.8 Reef released
by Yuri Weinstein 21 Mar '26

21 Mar '26

We're happy to announce the 8th, and expected to be last, backport release in the Reef series. We recommend that all users update to this release. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/ Known Issues ------------ * During QA for v18.2.8, it was found that there was a bug for upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a deprecated connection feature bit that was adopted to indicate a Reef OSD. This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded to Reef. There are no known issues associated with Pacific and Reef OSDs interoperating where Pacific OSDs are "advertising" Reef compatibility; however, out of an abundance of caution, we no longer recommend upgrading from Pacific to Reef directly. Security Fixes -------------- * CephFS Client: A fix was merged to prohibit unprivileged users from modifying the sgid or suid bits on a file. Previously, unprivileged users were inadvertently permitted to set these bits if they were the sole bits being modified. * Mgr Alerts: The SMTP SSL context was enforced in the mgr/alerts module to resolve a security vulnerability (GHSA-xj9f-7g59-m4jx). Notable Changes --------------- * RGW (RADOS Gateway): - Fixed an issue where bucket rm --bypass-gc was mistakenly removing head objects instead of tail objects, potentially causing data inconsistencies. - Fixed rgw-restore-bucket-index to handle objects with leading hyphens and to process versioned buckets correctly. - Addressed an issue in the msg/async protocol that caused memory locks and hangs during connection shutdown. - RGW STS: Made JWKS URL verification configurable for AWS compliance via the rgw_enable_jwks_url_verification configuration. * CephFS / MDS: - Prevented the MDS from stalling (up to 5 seconds) during rename/stat workloads by forcing the log to nudge for unstable locks after early replies. - Fixed cephfs-journal-tool so it no longer incorrectly resets the journal trim position during disaster recovery, which was causing stale journal objects to linger forever in the metadata pool. - Fixed a bug where ll_walk incorrectly processed absolute paths as relative paths. - Prevented the ceph fs volume create command from accidentally deleting user-created pools if the command aborted during cleanup. - MDS Batched Operations: Added a new mds_allow_batched_ops configuration option (default: true) to control whether the MDS can batch lookup or getattr RPCs. - CephFS Subvolumes: Added the ceph fs subvolume snapshot getpath command to allow users to retrieve the absolute path of a snapshot of a subvolume. * BlueStore: - Fixed a bug where the bytes_written_slow performance counter incorrectly reported 0 when using aio_write. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: efac5a54607c13fa50d4822e50242b86e6e446df

1 0

March Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 20 Mar '26

20 Mar '26

Hey folks, We will be having a Ceph science/research/big cluster call on Tuesday March 24th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: March 24th, 2026 9am Central US 14:00 UTC 3pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20260324 <https://pad.ceph.com/p/Ceph_Science_User_Group_20260324> Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Re: Recommended approach for hybrid devices
by Anthony D'Atri 20 Mar '26

20 Mar '26

Color me unamused. Reminds me of the Adaptec ACB 4000, which Sun's kernel detected by looking for an empty inquiry string. > On Mar 20, 2026, at 10:05 AM, Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > > I have encountered devices that claim to support multiple namespaces but do not implement the NVMe spec cleanly and send the same ID for each namespace. This causes the kernel to just show the first namespace.

1 0