- ceph-users - lists.ceph.io

Reflinks coming?
by Jay 27 Oct '26

27 Oct '26

I've been periodically checking on the reflink ticket[0] for a few years, and I just noticed that it's been updated with a target version of v22.0.0. Is that real? If so, is there anywhere I can read up on the implementation, plans, etc? 0 - <https://tracker.ceph.com/issues/1680> -- Jay ceph(a)tsuraan.net

3 3

20.2.4 Upgrade questions
by Jay 23 Oct '26

23 Oct '26

Looks like similar things are already being discussed on this list, but I'm not sure, so here goes. Q1 - I've updated all my service keys (mon, osd, etc) to the new type, and the "health detail" lines for AUTH_INSECURE_SERVICE_KEY_TYPE is gone, but after two days I still have the insecure rotating key messages: [WRN] AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE: 4 rotating auth service keys using insecure key types rotating service keys for mon using insecure key type: aes rotating service keys for mds using insecure key type: aes rotating service keys for osd using insecure key type: aes rotating service keys for mgr using insecure key type: aes I thought those would go away after a few hours? I stopped each process while I upgraded its keys, and I've since rebooted each host, but it looks like the old keys are still rotating or something? Is this a sign that something's gone wrong? Q2 - The guide [0] said it shouldn't be necessary to manually update the keyring files for the monitors. I assumed that meant the monitors would update their own keyrings, but they never did, so I did the import command for those as well. What was the expectation there? Should the keyrings have been rewritten, or do monitors now store their key info elsewhere? Q3 - OSD Lockbox? I saw the other thread and the linked script[1], so I think that answers my questions, but to be clear, a totally manual process would pretty much be: stop osd.{id} ceph auth rotate --key-type=aes256k client.osd-lockbox.{uuid} | tee keyring lvchange --addtag ceph.cephx_lockbox_secret={verbatim key from keyring} vg/lv start osd.{id} And that should pretty much do the trick? Starting the osd fires off ceph-volume activate, which rewrites the keyring in the osd tmpfs, so I don't think a person would need to do any manual file manipulation? Also a comment, I think the guide for updating OSDs is incorrect when the OSDs are dmcrypt. It says this: ceph-volume lvm list $OSD_ID --format json and find the lv_path field ceph-bluestore-tool --dev $DEV_PATH set-label-key --key osd_key -v $COPIED_KEYRING but when the OSD is encrypted, the lv_path refers to the encrypted block device, which doesn't have the bluestore metadata on it. What worked for me was the block_uuid field, and finding that entry under /dev/mapper, like this: # ceph-volume lvm list 0 --format json |grep '"ceph.block_uuid"' "ceph.block_uuid": "GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij", # ceph-bluestore-tool --dev /dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij show-label { "/dev/mapper/GB4MsC-lnvl-PaMW-xGwi-BpEb-eyUK-mu71Ij": { "osd_uuid": "013e9cce-66da-4b30-99a0-6e144665167e", ... Also the bit about set-label-key and $COPIED_KEYRING should be clear that it's going to write the value of $COPIED_KEYRING nto the bluestore metadata, so if you give it the value "keyring", then the stored ceph key will be "keyring". Grabbing the value from the "key = value" stanza of the keyring worked a lot better for me. 0 - <https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/#upgrad…> 1 - <https://codeberg.org/sur5r/ceph-osd-lockbox-key-rotate/src/branch/main/ceph…> -- Jay ceph(a)tsuraan.net

4 8

umbrella 21.2.0 QE validation status
by Yuri Weinstein 23 Oct '26

23 Oct '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76756#note-11 Release Notes - https://github.com/ceph/ceph/pull/70240 LRC upgrade - TBD Build 1 (umbrella-release 8ecd7c9aefe4cd9fecac47ea39487ed7700c1342): reruns of the failed/dead jobs are scheduled; some are still running. PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER! Seeking approvals/reviews for: smoke: Laura rados: Radek, Nitzan, Laura, Afreen (dashboard), Travis (rook) rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane, Guillaume rbd: Ilya krbd: Ilya nvmeof: PASSED upgrade: Laura, Patrick upgrade-clients:client-upgrade-tentacle-umbrella-umbrella: PASSED upgrade-clients:client-upgrade-squid-umbrella-umbrella: PASSED perf-basic: PASSED crimson-rados: Matan powercycle: PASSED TIA

1 1

BlueFS spilling over to SLOW even when DB has a lot of free space
by Jeremi-Ernst Avenant 22 Oct '26

22 Oct '26

Good day We run our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe containing 12 partitions. However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) has a lot of free space left. We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why? Even when we run `compact` against a cephfs_data HDD it goes away but the very next day it is back again. We run our cephfs_metadata on SSD disks and recently had our HPC ceph cluster fall over and received metadata corruption over 15 PB (from attempts to get it running again). (5 days after upgrading to 20.2.3 from 19.2.4). But what led to it was that there was a BlueFS spillover and the "META" went back onto the SSD (SLOW), without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools normally hover around 2-7% usage aggregated over all the SSDs. Below are some of our servers and the DB used / SLOW used. Server: B-01-01-cephosd HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB -- Server: B-01-03-cephosd HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB -- Server: B-01-05-cephosd HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za | www.ilifu.ac.za E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

3 5

EC plugin ISA-L production ready?
by Eugen Block 22 Oct '26

22 Oct '26

Hi *, since Tentacle the new default EC plugin is ISA-L, I wanted to ask if anyone is using it in production (maybe for several years already)? I searched tracker for issues with the cauchy technique and found several (haven't read all of them), sometimes in combination with FastEC (which isn't safe to use yet from my understanding), but for example this doesn't mention the technique: https://tracker.ceph.com/issues/74943 and it's not backported to Tentacle yet. So my current impression is that ISA may not be fully production ready, is it? Would it be safer to stick to jerasure for now although it's not being maintained anymore? One specific EC profile I'm dealing with is k=5 m=7 which can be created with "isa + cauchy" or "jerasure + reed_sol_van" but is refused by "isa + reed_sol_van". I'd appreciate any comments! Thanks! Eugen

2 2

Ceph User / Dev Meetup Tomorrow!
by Anthony Middleton 20 Oct '26

20 Oct '26

Hey Ceph friends, The next Ceph User + Dev Meetup is tomorrow, Wednesday, Oct. 21, at 7:00 AM PDT / 14:00 UTC. The agenda is still open, so bring your questions and topics to help move the conversation forward. RSVP here: https://t.ly/Oct-Ceph-UersDev See you there! Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Creating unfound objects on purpose
by Thomas Erbesdobler 19 Oct '26

19 Oct '26

Hi all, do you know about a reliable way to create unfound objects? I recently had a couple unfound objects on a production cluster after a group of osds crashed, and I would like to reproduce this state and experiment with it in my development environment. But therefore I first need to get unfound objects ;-). In all cases I've used ceph 19.2.3 with bluestore OSDs. What I tried so far: Attempt 1: * create a replicated pg with 3 shards, min_size=1 (same as in the production system at the time of the crash) * stop all but one osd, set norecover and nobackfill * write to a part of an object * start a second osd, wait until it boots and peers, stop the first osd * the object is indeed unfound, but I cannot run `ceph pg ... mark_unfound_lost ...` on it, since the osds who might have the object have not been queried * but when I destroy and recreate the first osd, the object becomes available again after recovery and reads the new content ~> recovery must have happened in some way Attempt 2: * same pg as in attempt 1 * stop all but one osd * write a lot to the pg (or rather its pool, that is, to create the data I've actually used the pool for rbd) * start one more osd of the pg and wait until recovery starts, then stop the first osd * the pg goes down, but no unfound objects have been created Basically I tried to follow the sequence sketched in https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#unf… . So, I guess it is more complicated to create unfound objects, maybe one requires to cause the "correct" osd crashes such that the pg's log somehow gets out of sync with the actual data stored on the osd? Any ideas would be welcome. Regards, Thomas

3 2

Oct Ceph Tech Talk
by Anthony Middleton 16 Oct '26

16 Oct '26

Hey Ceph friends, Just a reminder that our next Ceph Tech Talk is Monday, Oct 19th, at 12 pm EDT. The session will also feature four live demos covering Ceph RGW, Terraform/OpenTofu, S3 access, Kubernetes pod identity, and integration with VEXXHOST Inc.’s <https://www.linkedin.com/company/vexxhost/posts/?feedView=all> Sunrise OpenStack dashboard. RSVP and join the session here: https://www.linkedin.com/events/cephtechtalk-oidcwebidentityfor751324473649… Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

help me understand "slow heartbeats" warnings
by Janne Johansson 16 Oct '26

16 Oct '26

I see this at times, 2026-10-15T06:33:57.184296+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 3038.405ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:33:57.184319+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 3038.335ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:07.332951+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2981.937ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:07.332975+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2982.992ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:35.591255+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 2038.587ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:35.591280+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 2039.430ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:50.182523+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1284.844ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:50.182545+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1284.781ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:34:59.363750+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1228.156ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:34:59.363774+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1228.295ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:07.363735+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on back (longest 1220.582ms) (OSD_SLOW_PING_TIME_BACK) 2026-10-15T06:35:07.363771+0000 mon.mon-1 [WRN] Health check update: Slow OSD heartbeats on front (longest 1220.705ms) (OSD_SLOW_PING_TIME_FRONT) 2026-10-15T06:35:21.546156+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_BACK (was: Slow OSD heartbeats on back (longest 1220.582ms)) 2026-10-15T06:35:21.546210+0000 mon.mon-1 [INF] Health check cleared: OSD_SLOW_PING_TIME_FRONT (was: Slow OSD heartbeats on front (longest 1220.705ms) These lines are 10-15-20 seconds apart, and in that time span, the "longest" is lowering by .. a few hundred milliseconds. So over the course of two minutes something has gone from 3000 ms to under 1000. Now if this was a queue then it seems weird that it did not sink slower, assuming that the involved parts are working off items on the queue for minutes. If they were NOT working off the queue, one would assume it would grow by one second per second, like when you get 1 ops are blocked > 3276 sec on osd.21 and nothing is improving the situation, then that delay does grow by the second, totally reasonable. But the slow ops feels like a mystery to me, how it can be so specific in milliseconds while often working itself downwards so slowly that it takes literal minutes to get down below the warning limit. So what does this "Slow OSD heartbeats" actually measure at the time of these log lines? I often get them during expansion of the cluster or rolling OSD restarts, so its not necessarily even a strict error condition either. -- May the most significant bit of your life be positive.

2 4

S3 Select usage ahead of proposed deprecation
by Matthew H 15 Oct '26

15 Oct '26

Hi Team, PR #72337<https://github.com/ceph/ceph/pull/72337> proposes deprecating S3 Select in RGW and disabling it by default. Before we consider merging it, is anyone currently using this feature or aware of users who rely on it? If so, please share your use cases so we can understand the impact. Thanks!

2 2