- ceph-users - lists.ceph.io

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 03 Dec '25

03 Dec '25

On 2025-12-03 01:53, Michael Worsham via ceph-users wrote: > What does it show if you do 'ceph versions'? "ceph version 20.3.0-2957-g62bcf65e (62bcf65e8c0995783bb3e368909716346874ad62) tentacle (dev - RelWithDebInfo)" for all daemons. > Also, there is a way that you can upgrade individual daemons via "staggered" upgrade... > > "ceph orch upgrade --daemon-types <type1,Type2, etc.>" I have in fact already tried that, initially limiting the upgrade to mgr and mon daemons. Same as without the restriction - with "--ceph-version 20.2.0" the upgrade fails immediately with "ceph cannot downgrade to a dev release", with "--image quay.io/ceph/ceph:v20.2.0" the "cannot downgrade" appears in the upgrade status. -- MS

1 0

Migrating krbd -> librbd
by Boris 03 Dec '25

03 Dec '25

Hey, we are thinking to migrate from krbd to librbd. Anybody using librbd in larger setups? Any things we should consider on the storage nodes if we want to have 10k VMs via librbd?

1 0

500 - Internal Server Error ceph RGW Dashboard
by GLE, Vivien 03 Dec '25

03 Dec '25

Hi, I'm trying S3 on my cluster with a RGW in VM. My cluster is currently in v20.2.0 and RGW is in v19.2.1. S3 is working fine the only problem is that if I try to search (via RGW VM) for anything user relative (bucket or user management) on the object ceph dashboard this search leads to 500 - internal server error with those logs in the mgr 2025-12-03T09:47:06.545+0000 700619b22640 0 [dashboard INFO request] [::ffff:172.16.112.15:60786] [GET] [500] [0.155s] [admin] [513.0B] /api/rgw/user/dashboard 2025-12-03T09:47:06.585+0000 70061a323640 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 929, in get return self._get(uid, daemon_name=daemon_name, stats=stats) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 941, in _get result['uid'] = result['full_user_id'] KeyError: 'full_user_id' I found a workaround with deploying a container RGW via cephadm v20.2.0 and using it to manage my bucket and user. Does radosgw needs to be at the same version as the ceph cluster ? Or is it a bug ? Thanks Vivien

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 03 Dec '25

03 Dec '25

What does it show if you do 'ceph versions'? Also, there is a way that you can upgrade individual daemons via "staggered" upgrade... "ceph orch upgrade --daemon-types <type1,Type2, etc.>" Example types: mgr, mon, crash, osd, mds, rgw, rbd-mirror, and cephfs-mirror NOTE: Enforced upgrade order is: mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -- Michael ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, December 2, 2025 6:46 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. Hello, We have got a cephadm-managed test Ceph cluster created with a Tentacle Release Candidate, RC1 if memory serves me right. With 20.2.0 having officially been releases mid-November, we would ideally like to bring this cluster into production without data loss - none of the data already present on the OSDs is irreplaceable but we would prefer to retain snapshot history we have established during the test phase. Unfortunately, it turns out that the naive approach of having "ceph orch upgrade" upgrade the cluster to a production release does not work - if we run it with "--ceph-version 20.2.0" it immediately fails with the error "cannot downgrade to a dev version", "--image quay.io/ceph/ceph:v20.2.0" on the other hand appears to start successfully but gets immediately paused, with the same error message appearing in the upgrade status report. The "downgrade" bit is likely related to the fact that according to "ceph versions", the Ceph version we have presently got running is 20.3.0-2957-g62bcf65e (despite the fact the cephadm package we used clearly having been labelled 20.1.something). The "to a dev version" bit, however, has got me stumped given the long-established Ceph has been using the "even minor versions are production" convention for ages. Any thoughts on how we can get this cluster upgraded to a production release without wiping everything and starting from scratch? Thanks in advance! PS. cephadm on the admin node has already been upgraded to 20.2.0. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 02 Dec '25

02 Dec '25

Hello, We have got a cephadm-managed test Ceph cluster created with a Tentacle Release Candidate, RC1 if memory serves me right. With 20.2.0 having officially been releases mid-November, we would ideally like to bring this cluster into production without data loss - none of the data already present on the OSDs is irreplaceable but we would prefer to retain snapshot history we have established during the test phase. Unfortunately, it turns out that the naive approach of having "ceph orch upgrade" upgrade the cluster to a production release does not work - if we run it with "--ceph-version 20.2.0" it immediately fails with the error "cannot downgrade to a dev version", "--image quay.io/ceph/ceph:v20.2.0" on the other hand appears to start successfully but gets immediately paused, with the same error message appearing in the upgrade status report. The "downgrade" bit is likely related to the fact that according to "ceph versions", the Ceph version we have presently got running is 20.3.0-2957-g62bcf65e (despite the fact the cephadm package we used clearly having been labelled 20.1.something). The "to a dev version" bit, however, has got me stumped given the long-established Ceph has been using the "even minor versions are production" convention for ages. Any thoughts on how we can get this cluster upgraded to a production release without wiping everything and starting from scratch? Thanks in advance! PS. cephadm on the admin node has already been upgraded to 20.2.0. -- MS

1 0

Re: reef 18.2.8 QE validation status
by Afreen Misbah 01 Dec '25

01 Dec '25

Dashboard approved On Sat, Nov 29, 2025 at 10:47 PM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Rerun scheduled for krbd > > Still waiting for replies from dev leads > > On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> > > wrote: > > > > > > Note: this is an attempt to test and publish post EOL reef point > release > > > > > > Details of this release are summarized here: > > > > > > https://tracker.ceph.com/issues/73906#note-1 > > > Release Notes - TBD > > > LRC upgrade - N/A > > > > > > distro => jammy focal centos9 windows bookworm > > > > > > Seeking approvals/reviews for: > > > > > > smoke - Laura approved? > > > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? > > > (rados is also running on build 2 == build 1 + > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > > > rgw - Adam E approved? > > > > > > fs - Venky approved? > > > > > > orch - Adam King approved? > > > > > > rbd - Ilya approved? > > > krbd - Ilya approved? > > > > rbd approved, krbd needs a rerun as all jobs failed while > > going through cephlab.yml playbook (not related to RBD): > > > > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, > > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > > codepage or helper program, or other error.'] > > > > Thanks, > > > > Ilya > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Afreen

1 0

Test e-mail after updating TXT records
by Adam Kraitman 01 Dec '25

01 Dec '25

-- Adam Kraitman Systems Administrator Ceph Engineering IRC: akraitma

1 0

Removing a specific object.
by quaglio＠bol.com.br 01 Dec '25

01 Dec '25

Hi, I needed to remove a disk from a specific pool and now I'm facing a problem that I can't solve. There is a pool that has 1 "unfound" object. Furthermore, recovery is not happening for that PG because of this object. pg 3.792 is active+forced_recovery+recovery_unfound+undersized+degraded+remapped, acting [69], 1 unfound There is no problem removing this object. How do I delete it? Thanks Rafael.

1 1

Persistent “PGs not scrubbed in time” on Reef 18.2.7 cluster — seeking tuning guidance
by Zachary Wright-Faust 30 Nov '25

30 Nov '25

Hi all, I’m running a 10-node Ceph cluster (Reef 18.2.7) that has been having persistent issues keeping PG scrubs up to date as we migrate workloads from our previous production cluster. This environment provides CephFS only. The main CephFS data pool is an EC 4+2 pool backed by HDD OSDs, and the metadata pool is a REP 3 pool on SSD OSDs. The hardware is relatively old (mostly Dell PowerEdge R630s with SAS-2 attached JBODs). This cluster was deployed as an interim platform while we rebuild our production environment following unrecoverable MDS corruption. The scrub backlog started to accumulate as soon as we began migrating data onto this interim cluster, and although I’ve been able to mitigate it at times, it has been growing again now that the majority of our clients have shifted over to it. Below is the current cluster status: > cluster: > id: 150735d4-4449-11f0-a328-a0369ff049b4 > health: HEALTH_WARN > 126 pgs not scrubbed in time > > services: > mon: 5 daemons, quorum > sio-nautiloid01,sio-nautiloid02,sio-nautiloid03,sio-nautiloid07,sio-nautiloid06 > (age 4w) > mgr: sio-nautiloid02.jbayyi(active, since 5h), standbys: > sio-nautiloid01.ssmngh > mds: 1/1 daemons up, 1 standby > osd: 248 osds: 248 up (since 4w), 248 in (since 9w) > > data: > volumes: 1/1 healthy > pools: 6 pools, 8417 pgs > objects: 1.04G objects, 1.3 PiB > usage: 1.9 PiB used, 1.7 PiB / 3.6 PiB avail > pgs: 6913 active+clean > 669 active+clean+scrubbing+deep > 408 active+clean+scrubbing > 405 active+clean+snaptrim_wait > 22 active+clean+snaptrim > > io: > client: 37 MiB/s rd, 4.3 MiB/s wr, 1.11k op/s rd, 53 op/s wr > > Current scrub-related configuration: mgr advanced > osd_deep_scrub_interval 5184000.000000 > > mgr advanced > osd_scrub_max_interval 2419200.000000 > > mgr advanced > osd_scrub_min_interval 1209600.000000 > > osd advanced > osd_deep_scrub_interval 4838400.000000 > > osd advanced > osd_deep_scrub_randomize_ratio 0.150000 > > osd advanced > osd_max_scrubs 3 > > osd advanced > osd_scrub_chunk_max 25 > > osd advanced > osd_scrub_load_threshold 0.900000 > > osd advanced > osd_scrub_max_interval 2419200.000000 > > osd advanced > osd_scrub_min_interval 1209600.000000 > > osd advanced > osd_scrub_priority 10 > > osd advanced > osd_scrub_sleep 0.100000 > These intervals mirror what we had configured on our previous production cluster (Quincy 17.2.7). I introduced osd_scrub_sleep after observing high commit and apply latency in ceph osd perf. I’ve also experimented with higher scrub priority and a higher load threshold, but the number of overdue scrubs continues to grow. At this point I’m looking for guidance on: - Additional tuning options that might help scrub throughput - Whether my current interval choices are counterproductive - How to identify the primary bottlenecks (OSD load, disk latency, stride size, concurrency, etc.) Any insights would be greatly appreciated. Thanks, Zach

1 0

Re: cephadm without containers
by Anthony D'Atri 30 Nov '25

30 Nov '25

> Thus displacing the problem from developers to the admins as now admins have to maintain the whole container support environment, including the "slightly varying library packages etc." This whole exercise is pointless as Ceph cluster storage nodes are natively dedicated to Ceph There are LOTS of converged clusters, notably Proxmox and Rook within K8s. > thus the bare-metal machine is the real Ceph container. Just install the specific OS supported by Ceph Fleets don’t care for that approach. They don’t want to have to maintain automation and package and provisioning systems for multiple OSes, multiply the infosec burden, etc. With Podman and log-to-file, in addition to the cephadm orchestrator, one can still manage individual daemons with systemctl. > and you've got your Ceph "container" made out of metal. > > You can make an argument that monitor and/or manager daemons could run in containers to allow for migration to alternative nodes. Indeed, this is a big advantage. > Properly sized Ceph clusters come with a number of dedicated monitor nodes anyway In the Luminous days I felt that way, since mons were subtle and quick to anger. I liked to decouple them from OSDs, which were more likely to need node reboots. I no longer due. I’ve had a pre-cephadm cluster with 5x mon/RGW nodes, two of which broke, and it took me *18 months* to convince the TAC to fix them. The cluster was one more failure from an outage, where with a converged cluster the full quorum would be maintained automagically. Defining the service spec to abstract based on a “mon” host label, and judiciously applying that label to one or at most two nodes per rack / failure domain maintains prudent anti-affinity. > therefore migration usually means add a new monitor node to the cluster and remove the old one, thus making the monitor containers point moot, too. Adding a new node to the cluster is far from trivial when the DC is overseas and unstaffed. It would also mean having a warm / hot spare node, which with your reasoning would be a different spec than the OSD nodes. The fewer chassis types in a DC the better, from an interchangeability, servicing, and sparing perspective. And, yes, I’ve run fleets of as many as 15000 nodes. ymmv. — aad

1 0