- ceph-users - lists.ceph.io

Make backup to another server.
by Nmz 02 Jun '25

02 Jun '25

Hello everyone, I came to Ceph from ZFS world. In ZFS I can use snapshots as backup way. Keep it localy or send to another server/pool. Is it possible to do it the same using Ceph? Right now in my cluster I use RBD and CephFS. RBD data structure: Ceph pools pool proxmox_metadata Rep SSD pool proxmox_data EC SSD storage.cfg rbd: ceph_rbd content images data-pool proxmox_data pool proxmox_metadata Ceph pools pool proxmox_data_1_ssd_other_hdd R 1 SSD 2 HDD storage.cfg rbd: ceph_rbd_1ssd_hdd content images pool proxmox_data_1_ssd_other_hdd CephFS data structure: Ceph pools pool cephfs_metadata Rep SSD pool cephfs_data Rep SSD pool web_data EC SSD pool db_data EC SSD pool data EC HDD # ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data web_data db_data data ] subvolumegroup | data_pool | subvolume home | web_data | nmz .... db | db_data | sql persoanl | data | Video .... As you can see subvolumes use specific ceph pools. For every subvolume I use snapshots. Ceph backup server will work in cold mode, single node, not 24/7. What is the best way to send data to backup server ? I want to use incremental backup sync. Can I achieve it ? Thanks

2 1

Mon Election Leader goes unresponsive with 100% cpu usage on fn_monstore
by Wesley Dillingham 01 Jun '25

01 Jun '25

Tracker issue made here with some additional details: https://tracker.ceph.com/issues/71501 Cluster version 18.2.4 I came to assist with a non-functional cluster which had OSDs erroneously --force purged and led to multiple (6) degraded + inactive PGs (4 remaining shards in a 4+2) and 1 remapped+incomplete PG (3 shards in a 4+2) In an effort to restore order to the cluster and get backfill working: the degraded PGs shared a common primary OSD and that OSD was restarted. Additionally min_size was dropped from 5 to 4 for this pool (a temporary measure while the cluster recovered). This caused the inactive degraded PGs to go active and start their backfill. The PGs steadily worked on backfill for a few hours. Near immediately after the final of the 6 degraded PGs finished its backfill the monitor quorum broke and the cluster became unresponsive. In this state 2 of the 5 mons showed 100% cpu usage. In an attempt to fix the mon quorum some combination of monitor service restart attempts occurred and ultimately all ceph services were brought down in order to isolate the MON issue. As the situation stands currently any combination of starting 3 MONs (quorum eligible at 3) causes the lowest MON (the to-be-leader) to hit a 100% cpu with fn_monstore thread and render unresponsive the admin socket of that mon only (the other 2 respond via admin socket and show either probing or electing). I have captured logs of the to-be-leader mon (its daemon started first) with debug_mon = 20 and debug_ms = 20 (I should probably recapture with debug_paxos = 20). The pegged cpu only occurs once the third MON is started (seems to be an election issue). The MON has been allowed to run for hours with no progress in that state. Eventually it seems the leader goes into the (leader) state (Claims "I win" in the logs) but the other 2 mons continue their election cycle. In states of probing or electing still. I have verified NTP clock is fine and sync'd to same source on all the mons and connectivity between the mons at both ports is functional. The situation the PGs were in during the subsequent MON fault leads me to believe the problem is more complex than typical monitor election issues. Also of note the backing disk of the to-be-leader MON is mostly idle. At this point I am interested in taking backups of all the mon stores and injecting a modified single-mon monmap in a mon and seeing if we can get back up but am also concerned that that single mon will be the de-facto leader and also be unresponsive. Interested in any suggestions from the wider community. Thanks! Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes(a)wesdillingham.com

2 5

19.2.2: Warning,Smartctl has received an unknown argument (error code -22)
by Harry G Coin 31 May '25

31 May '25

Using 19.2.2, we notice under cluster/osds/'device health' on the dashboard, for all osds no matter the server: Warning Smartctl has received an unknown argument (error code -22). You may be using an incompatible version of smartmontools. Version >= 7.0 of smartmontools is required to successfully retrieve data. That error code resolves to 'unknown attribute' in the smartctl docs. However, the same result occurs whether the drive is HGST, Seagate, or Western Digital. "State of Health" is always "Stale" "Life Expectancy" is always "n/a> 6 weeks" Of course the 'diskprediction_local" module has been broken for over a year, as it requires a no-longer-distributed rev of a sub-package, but that shouldn't stop the smartctl command from normal operations. Any ideas? Thanks! Harry Coin

2 2

How does Ceph delete objects / data on disk?
by Stefan Kooman 29 May '25

29 May '25

Hi, I would like to get a better understanding at what happens under the hood when an object is deleted in a Ceph cluster. Is there a (detailed) write up of the deletion process? I would like to be able to answer questions like: - How can we be sure that after the successful completion of for example a "rbd rm $image" command it is impossible for the data to be restored. - How do we know we don't get deleted data returned that was marked as deleted (but never overwritten with new data) Can we be sure? Or are there ways to perform a recovery procedure (as a storage admin with access to all components of the cluster), i.e. gather information about higher level Ceph abstractions (rbd, file in CephFS, RGW bucket) that have existed in the past, gather information about what objects where in use, do CRUSH calculation (and consult upmaps) to obtain OSDs involved, Query RocksDB / Bluestore to obtain information about LBA on disk, read the data and in this way reconstruct the object(s). Pathological example: rbd rm $image (successful deletion) ceph pause immediately after that Do the recovery procedure noted above How likely is it that we would be able to recovery the data? I understand it's hard to put a number on it, so pointers to where (parts) of this process is described will help me getting a better understanding. I'm also aware that it highly depends on how busy a cluster would be, what type of media involved, if the discard option of Ceph and disks is in use, and many other factors. Thanks, Stefan

4 3

Adding OSD with separate DB via "ceph orch daemon add osd"
by Ryan Rempel 28 May '25

28 May '25

I'm expanding a small Ceph cluster from 4 nodes to 5 nodes. The new node is a bit more sophisticated than the others, since it has some SSD storage that I'd like to use for DB+WAL (which I haven't done before, it has just been rotational disks). I'm using cephadm for orchestration, and normally add osds via "ceph orch daemon add osd". I prefer to add the osds in this "manual" way (rather than "ceph orch apply" with a spec) mainly because my infrastructure is not uniform (for better or worse, I'm working with hardware that becomes available in different ways over time, as I gradually upgrade things and add things). Looking at this page: https://docs.ceph.com/en/squid/cephadm/services/osd/ ... it isn't entirely clear to me whether it's possible to specify a separate DB device when using the "ceph orch daemon add osd" procedure. There is a description of how to do it with a service spec, but how you would specify the DB device for "ceph orch daemon add osd" does not appear to be described. So, my first question is whether it's possible to specify a separate DB via "ceph orch daemon add osd"? If not, I'll need to explore the service spec approach. I suppose I can use the "unmanaged: true" option in the spec (to keep it as "manual" as possible). The remaining puzzle is how to use the SSD as DB+WAL for more than one OSD. At this point, the SSD is the raw device — I haven't done anything with it manually in LVM or whatever. In the service spec description above, I see that there is a "db_slots" key. So, I suppose that I could specify the "whole" SSD and provide for the number of slots? However, I don't necessarily want every slot to be the same size (because of my unfortunately heterogeneous hardware). So, I also see that there is a "block_db_size" and "block_wal_size". But it's unclear how this relates to "db_slots" — which one would determine how the SSD is sliced up? I'd actually be happy to pre-slice the SSD (e.g. with LVM) and then directly specify which SSD slice is the DB+WAL for which OSD, if that's a feasible approach. Though, I'd still be interested in knowing whether I need to set something for "block_db_size" and "block_wal_size", or whether it's enough to just actually make a certain size of LVM volume available for DB+WAL. Normally I'd just experiment, but that might be disruptive to the working cluster. I guess I could at least turn off rebalancing while I try things out? The other documentation I'm now reading is the documentation for ceph-volume, which appears to be related: https://docs.ceph.com/en/squid/ceph-volume/lvm/batch/ It mentions, for instance, things like db_slots and block_db_size. The implication is that db_slots is an alternative to block_db_size — that you wouldn't specify both, for instance. I'm also reading the ceph-volume docs for "prepare". I suppose if I find that more suitable, it might be possible to "prepare" and OSD with ceph-volume and then "adopt" it with cephadm? Well, just writing the email has given me a bit more clarity about things to try, but I'd certainly be happy for any guidance. Ryan Rempel Director of Information Technology Canadian Mennonite University

3 4

radosgw-admin: the RGW management nightmare
by Michel Jouvin 28 May '25

28 May '25

Hi, We have a Ceph cluster hosting ~10 RGW realms and in our experience, radosgw-admin is making the Ceph admins life very hard... The opposite experience to other Ceph management tools. In addition to the lack of verifications before applying modifications that leads to the issues describe in another thread were you can have a configuration that cannot be commited and at the same time cannot be fixed, I have the feeling that one of the major issue is that radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these 3 different kinds of objects as completely independent from the others. It is made worst by the fact that radosgw-admin defines a default object in each category, used implicitely (and slilently) if you don't select one specific object. The consequence of these 2 problems is that if you modify a zone, you can endup modifying the zonegroup of another realm than the one the zone belongs to (as there is no concept of a zone belonging to a zonegroup in radosgw-admin). It happens for example if you modify a zone (with zone modify) to make it the master zone (of a zonegroup in fact) and you don't specify the zonegroup the zone belongs to/is associated with: it will in fact update the default zonegroup that can be in another realm or at least not be the zonegroup the zone is associated with. A less impactful but nevertheless unexpected result is if you do something like: 'zonegroup list --rgw-realm my_realm'. You would expect to see only the zonegroups associated with the realm you specified where you'll get the list of all the zonegroups defined. I can imagine that chaning this is difficult without breaking backward compatibility. IMO, an improvement would be the ability not to have a default realm, zonegroup, zone. At least you would have an error if you don't specify explicitely the object to modify. Am I the only one struggling with radosgw-admin? Is there any plan to improve this? Best regards, Michel

3 3

[ceph-user] ceph-ansible OS Compatibility Matrix.
by Danish Khan 28 May '25

28 May '25

Dear Team, We have few ceph-ansible clusters which are still running on Ubuntu 18 and 20. We want to upgrade OS to ubuntu 22. But my ceph clusters are still running on octopus version. Is there any document which can suggest if I can upgrade OS first or do I need to upgrade ceph major version first. Regards, Danish Khan

2 1

Not using image 'xxx' as it's not in list of non-dangling images with ceph=True label
by Patrick Begou 27 May '25

27 May '25

Hi Ceph community, I'm deploying a fresh Ceph cluster using Almalinux9 and Squid version. Deployment with: "dnf -y install https://download.ceph.com/rpm-19.2.2/el9/noarch/cephadm-19.2.2-0.el9.noarch…" Then bootstrap (all runs fine). But launching cephadm shell I get this message that I cannot fully understand from podman: "Not using image 'xxx' as it's not in list of non-dangling images with ceph=True label" I saw that I can clean old images with "podman system prune -a" but this is a fresh install. Can I continue safely to deploy or is this a problem ? Thanks Patrick

2 1

CEPH Reef - HDD with WAL and DB on NVME
by Steven Vacaroaia 27 May '25

27 May '25

Hi, I am a bit confused by the documentation as it seems to say that, when using bluestore, there is no need to put the journaling on a faster storage when creating OSD https://docs.ceph.com/en/reef/rados/configuration/osd-config-ref/ <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.ceph…> But then I found this : “… BlueStore is deployed across one or two additional devices: - A *write-ahead log (WAL) device* (identified as block.wal in the data directory) can be used to separate out BlueStore’s internal journal or write-ahead log. Using a WAL device is advantageous only if the WAL device is faster than the primary device (for example, if the WAL device is an SSD and the primary device is an HDD). - A *DB device* (identified as block.db in the data directory) can be used to store BlueStore’s internal metadata. BlueStore (or more precisely, the embedded RocksDB) will put as much metadata as it can on the DB device in order to improve performance. If the DB device becomes full, metadata will spill back onto the primary device (where it would have been located in the absence of the DB device). Again, it is advantageous to provision a DB device only if it is faster than the primary device. …” Is it still recommended to put WAL and DB on a faster disk , right ? How can this be done with cephadm .? Is there a documented procedure for fixing this if it was not done from the beginning ? Many thanks Steven

1 0

v17.2.9 Quincy released
by Yuri Weinstein 27 May '25

27 May '25

This is the ninth backport (hotfix) release in the Quincy series. We recommend that all users update to this release. https://ceph.io/en/news/blog/2025/v17-2-9-quincy-released/ Please be aware that this hotfix was finalized after its end-of-life (EOL) declaration and subsequent removal from the build system. As a result, container builds were not created, and the hotfix has only been subject to limited testing. Users should expect to see the el8 rpm subdirectory empty and the "dnf" commands are expected to fail with 17.2.9. They can choose to use 17.2.9 RPM packages for centos 8/el8 provided by CERN as a community member or stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 These CERN packages come with no warranty and have not been tested. The software in them has been tested by Ceph according to `platforms <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms>`_ . The repository for el8 builds is hosted by CERN on `Linux@CERN <https://linuxsoft.cern.ch/repos/ceph-ext-quincy8el-stable/>`_ . The public part of the GPG key used to sign the packages is available at `RPM-GPG-KEY-Ceph-Community <https://linuxsoft.cern.ch/repos/RPM-GPG-KEY-Ceph-Community>`_ . Also note that Ceph now builds against OpenSSL 3.5.0, which may affect EL package users who are on distros that still reference an older version. Notable Changes --------------- This release fixes a critical BlueStore regression in versions 17.2.8 (#63122). Users running this release are recommended to upgrade at their earliest convenience. Changelog --------- - [quincy] os/bluestore: fix \_extend\_log seq advance ([pr#63122](https://github.com/ceph/ceph/pull/63122), aclamk) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.9.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: f817ceb7f187defb1d021d6328fa833eb8e943b3

1 0