- ceph-users - lists.ceph.io

Ceph Tentacle release timeline — when?
by Gregory Farnum 07 Feb '25

07 Feb '25

Hi all, We in the Ceph Steering Committee are discussing when we want to target the Tentacle release for, as we find ourselves in an unusual scheduling situation: * Historically, we have targeted our major release in early Spring. I believe this was initially aligned to the Ubuntu LTS release. (With cephadm and containerization I'm not sure this is particularly relevant any more?) * Unfortunately, due to issues in the sepia lab and with testing infrastructure, we had big slips on Reef (August), and we tried to get Squid back on track but it ended up slipping farther (September). * The sepia lab needs to physically move this year out of the current Red Hat community hosting area in to a new lab (hosted by IBM). We're not sure how disruptive this will be, but it will definitely require changes as our current tests are somewhat dependent on the shape of the underlying hardware, which is changing. * There are not many major features available yet. Major upcoming features include the fast EC code, userspace and management support for cephfs encryption, and RBD mirroring groups. Different parts of these are coming throughout the calendar year but we don't have firm timetables yet. We have basically discarded attempting to do an early spring release, since there doesn't seem to be much point. The obvious options are 1) Do a fall release, targeting 12 months after Squid (so, late September, with a freeze in early July). 2) Wait until early spring 2026, with a freeze in January. (This doesn't preclude other options as we gather more input, but we are starting with these for discussion.) We are soliciting feedback on the impact these options will have on our various downstreams and direct upstream consumers. The CSC's *early and tentative* preference is for spring 2026, for the following reasons: 1) the sepia lab move is going to involve an unknown amount of development, and we are gunshy after issues the last couple years. 2) While there is feature work in development, we are unsure how much of interest will be both ready to freeze by mid-summer, and not suitable for backport to our stable releases. 3) Fall releases seem awkward with many organizations wanting to freeze infrastructure over the holidays. 4) While the data is conflicting, it seems like deployments are lagging more on major-version upgrades and so a longer release cycle may make future updates easier for everyone. If the Ceph release schedule has a large impact on your organization or planning, please let us know your thoughts and forward to your downstreams. :) Thanks! -The Ceph Steering Committee, by way of Greg

13 13

v19.2.1 Squid released
by Yuri Weinstein 07 Feb '25

07 Feb '25

We're happy to announce the 1st backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-1-squid-released/ Notable Changes --------------- * CephFS: The command `fs subvolume create` now allows tagging subvolumes by supplying the option `--earmark` with a unique identifier needed for NFS or SMB services. The earmark string for a subvolume is empty by default. To remove an already present earmark, an empty string can be assigned to it. Additionally, the commands `ceph fs subvolume earmark set`, `ceph fs subvolume earmark get`, and `ceph fs subvolume earmark rm` have been added to set, get and remove earmark from a given subvolume. * CephFS: Expanded removexattr support for CephFS virtual extended attributes. Previously one had to use setxattr to restore the default in order to "remove". You may now properly use removexattr to remove. You can also now remove layout on the root inode, which then will restore the layout to the default. * RADOS: A performance bottleneck in the balancer mgr module has been fixed. Related Tracker: https://tracker.ceph.com/issues/68657 * RADOS: Based on tests performed at scale on an HDD-based Ceph cluster, it was found that scheduling with mClock was not optimal with multiple OSD shards. For example, in the test cluster with multiple OSD node failures, the client throughput was found to be inconsistent across test runs coupled with multiple reported slow requests. However, the same test with a single OSD shard and with multiple worker threads yielded significantly better results in terms of consistency of client and recovery throughput across multiple test runs. Therefore, as an interim measure until the issue with multiple OSD shards (or multiple mClock queues per OSD) is investigated and fixed, the following change to the default HDD OSD shard configuration is made: - `osd_op_num_shards_hdd = 1` (was 5) - `osd_op_num_threads_per_shard_hdd = 5` (was 1) For more details, see https://tracker.ceph.com/issues/66289. * mgr/REST: The REST manager module will trim requests based on the 'max_requests' option. Without this feature, and in the absence of manual deletion of old requests, the accumulation of requests in the array can lead to Out Of Memory (OOM) issues, resulting in the Manager crashing. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 58a7fab8be0a062d730ad7da874972fd3fba59fb

2 1

RGW issue, lost bucket metadata ?
by Cyril Duval 06 Feb '25

06 Feb '25

Hello folks ! This is my first time posting here, sorry in advance if I miss some of the etiquette. I'm facing some issues with RGW and especially the radosgw-admin tool, on what appears to be lost bucket metadata, or maybe corruption of some elements of the buckets. I have opened a ticket on https://tracker.ceph.com/issues/69818 where I put more details, but in summary I have the `failure: (2) No such file or directory:` kind of error on most radosgw-admin commands interacting with these buggy buckets. Lifecycle policies seem to be affected too, as I have a `lifecycle: RGWLC::process() failed to get obj head lc.5, ret=-5` error. I'm running Ceph 18.2.2 there. Is it something familiar to anyone ? Or does anyone know where I can have a look further to better understand the issue and solve it ? I tried giving a shot at the cpp code, but unfortunately I'm no expert there, I'm lost. Thanks in advance, -- Cyril Duval cyril.duval(a)diabolocom.com

1 0

Best way to add back a host after removing offline - cephadm
by Kirby Haze 06 Feb '25

06 Feb '25

I have a 3 node test 17.2.7 cluster, and I decided to power one of the hosts down which contained a mon, 6 osds and a standby mgr. Then I used the `ceph orch host rm <host> --force --rm` to remove the host (after powering it down). All of this looks expected except the logs after removing the host. These pools have size 3 min_size 2. Actually I'm unsure what would happen if I were even to successfully add back the host as those objects would be degraded. Besides the error from cephadm, what is ceph thinking when I pulled the plug then removed that host? Am I guaranteed some type of recovery if I do successfully add back that host? ---- Before removing host root@ceph-test-2:/# ceph -s cluster: id: fca870d8-e431-11ef-8000-bc2411363b7d health: HEALTH_OK services: mon: 3 daemons, quorum ceph-test-2,ceph-test-3,ceph-test-4 (age 29m) mgr: ceph-test-2.vbjhdq(active, since 31m), standbys: ceph-test-4.jjubsa osd: 18 osds: 18 up (since 22m), 18 in (since 28m) rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 5 pools, 129 pgs objects: 223 objects, 581 KiB usage: 5.2 GiB used, 8.8 TiB / 8.8 TiB avail pgs: 129 active+clean root@ceph-test-2:/# ceph orch host ls HOST ADDR LABELS STATUS ceph-test-2 10.0.0.52 _admin,rgw ceph-test-3 10.0.0.53 ceph-test-4 10.0.0.54 ------------- After removing host # ceph orch host rm ceph-test-4 --offline --force Removed offline host 'ceph-test-4' # ceph orch ps mon.ceph-test-4 ceph-test-4 stopped osd.1 ceph-test-4 error osd.7 ceph-test-4 error .... .... # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.85956 root default -5 2.92978 host ceph-test-2 2 hdd 0.48830 osd.2 up 1.00000 1.00000 4 hdd 0.48830 osd.4 up 1.00000 1.00000 8 hdd 0.48830 osd.8 up 1.00000 1.00000 11 hdd 0.48830 osd.11 up 1.00000 1.00000 14 hdd 0.48830 osd.14 up 1.00000 1.00000 16 hdd 0.48830 osd.16 up 1.00000 1.00000 -3 2.92978 host ceph-test-3 0 hdd 0.48830 osd.0 up 1.00000 1.00000 3 hdd 0.48830 osd.3 up 1.00000 1.00000 6 hdd 0.48830 osd.6 up 1.00000 1.00000 9 hdd 0.48830 osd.9 up 1.00000 1.00000 12 hdd 0.48830 osd.12 up 1.00000 1.00000 15 hdd 0.48830 osd.15 up 1.00000 1.00000 root@ceph-test-2:/rootfs/root# ceph -s cluster: id: fca870d8-e431-11ef-8000-bc2411363b7d health: HEALTH_WARN 6 failed cephadm daemon(s) Degraded data redundancy: 145/669 objects degraded (21.674%), 24 pgs degraded, 71 pgs undersized services: mon: 2 daemons, quorum ceph-test-2,ceph-test-3 (age 50m) mgr: ceph-test-2.vbjhdq(active, since 2h), standbys: ceph-test-3.wzmioq osd: 12 osds: 12 up (since 51m), 12 in (since 2h); 58 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones) data: pools: 5 pools, 129 pgs objects: 223 objects, 581 KiB usage: 3.5 GiB used, 5.9 TiB / 5.9 TiB avail pgs: 145/669 objects degraded (21.674%) 75/669 objects misplaced (11.211%) 54 active+clean+remapped 47 active+undersized 24 active+undersized+degraded 4 active+clean progress: Global Recovery Event (50m) [================............] (remaining: 37m) The cephadm logs show this as well 2025-02-06T06:27:46.936+0000 7f85026a4700 -1 log_channel(cephadm) log [ERR] : auth get failed: failed to find osd.7 in keyring retval: -2 Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1002, in _check_daemons self.mgr._daemon_action(daemon_spec, action=action) File "/usr/share/ceph/mgr/cephadm/module.py", line 2136, in _daemon_action daemon_spec) File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 494, in generate_config extra_ceph_config=daemon_spec.ceph_conf) File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 520, in get_config_and_keyring 'entity': entity, File "/usr/share/ceph/mgr/mgr_module.py", line 1593, in check_mon_command raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr} retval: {r.retval}') mgr_module.MonCommandFailed: auth get failed: failed to find osd.7 in keyring retval: -2

2 1

EC pool - Profile change
by Devender Singh 06 Feb '25

06 Feb '25

Hello all Is there a way to change ec pool profile from host to osd? Regrds Dev

5 7

Backfills Not Increasing.
by Devender Singh 05 Feb '25

05 Feb '25

Hello all I tried disabling osd_mclock_override_recovery_settings to true and increased backfill from 1 to 15 after disabling scrub and deep-scrub. There was no change I started from 1 then 3 then 5 then 7 then 10 then 13 and 15. But no change. So it a default behaviour or bug? # ceph config get osd osd_mclock_override_recovery_settings true # ceph config get osd osd_max_backfills 15 data: volumes: 2/2 healthy pools: 20 pools, 4065 pgs objects: 21.55M objects, 80 TiB usage: 209 TiB used, 283 TiB / 493 TiB avail pgs: 3912380/77920842 objects misplaced (5.021%) 3873 active+clean 168 active+remapped+backfill_wait 24 active+remapped+backfilling Regards Dev

1 0

CephFS subdatapool in practice?
by Otto Richter (Codeberg e.V.) 05 Feb '25

05 Feb '25

Hello everyone, I am maintaining a cost-optimized Ceph cluster at Codeberg.org, that is serving a lot of small random IO (Git operations). Currently, the data is stored in a single CephFS pool on mixed SSD and HDD cluster, where the HDDs have a primary affinity of 0 to shift the reads to the SSDs. In a quest to further optimize our setup, I discovered the subdatapool feature described at https://github.com/TheJJ/ceph-cheatsheet#subdatapool (assigning multiple pools to a filesystem and using setfattr to allow writes in a certain folder to a separate pool). It feels like this would allow us to cut cost by moving certain directories with large files to a cheaper pool than that used for Git operations easily, without doing data migrations such as splitting data into multiple filesystems. Is anyone using this feature in production and can comment on it? Are there best practices regarding the usage of this feature? Any caveats to be aware of, such as overhead, latency, long-term maintainability? Since certain "interesting" features have been deprecated in the past, such as inline_data, I thought it would be better to ask here for feedback first. Thank you for the comments and have a nice day! Kind Regards Otto -- https://codeberg.org Codeberg e.V. – Arminiusstraße 2-4 – 10551 Berlin – Germany Registered at registration court Amtsgericht Charlottenburg VR36929.

3 2

Re: cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 05 Feb '25

05 Feb '25

Hello, I ran the commands from the "regular" cephadm shell ('cephadm shell --') but did not specify an OSD (I wasn't aware you could do that) When you ran the ceph-volume command, did you do it within the container of the OSD? Running it with the container of the OSD would make sense as to why I was getting the errors I was getting. I can give it a try and let you know how it goes, and if necessary go from there. Sent from my mobile device. Please excuse brevity and ttpos.

2 19

Merge DB/WAL back to the main device?
by Jan Kasprzak 05 Feb '25

05 Feb '25

Hi all, while reading a sibling thread about moving DB/WAL to a separate device, I wonder whether is it possible to go the other way round as well, i.e. to remove a metadata device from an OSD and merge metadata back to the main storage? What I am trying to do: My OSD nodes are 1U boxes with 4 drive bays, two of which support NVMe. They have two small-ish NVMe drives and two large HDDs. On NVMe, there is a partition for an OS (configured as RAID-1 across both NVMe drives), and the rest of each NVMe is used as metadata for one of the HDD-based OSDs. So I have two OSDs per node. Now I am considering replacing those small NVMes with much larger ones, and use part of them for an OS as before, part of them as metadata for the HDD-based OSDs as before, and the rest as new NVMe-only OSDs. Can this be done without tearing the original HDD+NVMe-based OSDs down and recreating them again? Being able to remove a metadata device from the OSD would help in this case. Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

4 8

Spec file question: --dry-run not showing anything would be applied?
by Alan Murrell 05 Feb '25

05 Feb '25

Sorry, I just got on this list and am asking a number of questions… ☹ It was suggested in one of my other threads to create and apply a spec file that would have cephadm put the DB for rotational drives onto SSD when it creates the OSD on those drives. I came up with he following spec file: --- START --- service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 model: 'Samsung SSD 990 PRO 1TB' limit: 1 --- service_type: osd service_id: osd_spec_ssd placement: host_pattern: '*' data_devices: model: 'KINGSTON SEDC1500M1920G' --- END --- But when I run ‘ceph orch apply osd -i ./osd_spec.yml --dry-run’, I get the following output: --- START --- WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any of these conditions change, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+-------------+ +---------+------+--------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ --- END --- Which seems to indicate that nothing would be applied, but I am not sure if that is because my spec file is wrong or if it is because my rotational drives already have OSDs (and I have migrated their DB/WAL to the SSDs)?

3 2