- ceph-users - lists.ceph.io

ceph quincy cannot change osd_recovery_max_active, please help
by 辣条➀号 02 Feb '23

02 Feb '23

Greetings from the enthusiastic ceph official team! Recently, when our company was using ceph quincy (stable), it was found that the initial value of osd_recovery_max_active could not be changed. 1. When I try to modify osd_mclock_override_recovery_settings to true, I get an error. It seems that there is no such option? How should I modify the initial value of osd_recovery_max_active root@pve-ceph01:~# ceph config set osd osd_mclock_override_recovery_settings true Error EINVAL: unrecognized config option 'osd_mclock_override_recovery_settings'

1 0

CEPHADM_STRAY_DAEMON does not exist, how do I remove knowledge of it from ceph?
by ceph＠mikesoffice.com 01 Feb '23

01 Feb '23

Hi All, I'm getting this error while setting up a ceph cluster. I'm relatively new to ceph, so there is no telling what kind of mistakes I've been making. I'm using cephadm, ceph v16 and I apparently have a stray daemon. But it also doesn't seem to exist and I can't get ceph to forget about it. $ ceph health detail [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon mon.cmon01 on host cmgmt01 not managed by cephadm mon.cmon01 also shows up in dashboard->hosts as running on cmgmt01. It does not show up in the monitors section though. But, there isn't a monitor daemon running on that machine at all (no podman container, not in process list, not listening on a port). On that host in cephadm shell, # ceph orch daemon rm mon.cmon01 --force Error EINVAL: Unable to find daemon(s) ['mon.cmon01'] I don't currently have any real data on the cluster, so I've also tried deleting the existing pools (except device_health_metrics) in case ceph was connecting that monitor to one of the pools. I'm not sure what to try next in order to get ceph to forget about that daemon.

3 2

How to get RBD client log?
by Jinhao Hu 01 Feb '23

01 Feb '23

Hi, How can I collect the logs of the RBD client?

3 3

CLT meeting summary 2023-02-01
by Casey Bodley 01 Feb '23

01 Feb '23

distro testing for reef * https://github.com/ceph/ceph/pull/49443 adds centos9 and ubuntu22 to supported distros * centos9 blocked by teuthology bug https://tracker.ceph.com/issues/58491 - lsb_release command no longer exists, use /etc/os-release instead - ceph stopped depending on lsb_release in 2021 with https://github.com/ceph/ceph/pull/42770 * ubuntu22 not blocked by teuthology, but the new python version breaks most of the rgw tests can we drop centos8 or ubuntu20 support for reef? * we usually support the latest centos and two ubuntu LTSs * users need an upgrade path that doesn't require OS and ceph upgrade at the same time * we might be able to drop centos8 support for Reef by adding centos9 support to Quincy * python versioning issues make longer-term support of older distros problematic. related work: - https://github.com/ceph/ceph/pull/41979 - https://github.com/ceph/ceph/pull/47501 ondisk format changes in minor releases * https://github.com/ceph/ceph/pull/48915 introduced some BlueFS log changes in 16.2.11 which makes it incompatible with previous Pacific releases. Hence no downgrade is permitted any more. - doc text tracked in https://tracker.ceph.com/issues/58625 * how do we prevent these issues in the future? - better testing of mixed-version rgw/mds/mgr/etc infrastructure update * a planned network outage yesterday still affecting LRC

1 0

Adding Labels Section to Perf Counters Output
by Ali Maredia 01 Feb '23

01 Feb '23

Hi Ceph Developers and Users, Various upstream developers and I are working on adding labels to perf counters (https://github.com/ceph/ceph/pull/48657) We would like to understand the ramifications of changing the format of the json dumped by the `perf dump` command for the Reef Release on users and components of Ceph. As an example given in the PR, currently unlabeled counters are dumped like this in comparison with their new labeled counterparts. "some unlabeled_counter": { "put_b": 1048576, }, "some labeled_counter": { "labels": { "Bucket: "bkt1", "User: "user1", }, "counters": { "put_b": 1048576, }, }, Here is an example given in the PR of the old style unlabeled counters being dumped in the same format as the labeled counters: "some unlabeled": { "labels": { }, "counters": { "put_b": 1048576, }, }, "some labeled": { "labels": { "Bucket: "bkt1", "User: "user1", }, "counters": { "put_b": 1048576, }, }, Would users/consumers of these counters be opposed to changing the format? Why is this the case? As far as I know there are ceph-mgr modules related to Prometheus and telemetry that are consuming the current unlabeled counters. Also this topic will be discussed at the upcoming Ceph Developer Monthly EMEA as well. Best, Ali

2 1

Documentation - February 2023 - Request for Comments
by John Zachary Dover 01 Feb '23

01 Feb '23

This is an email thread for reporting documentation concerns. I won't guarantee that our documentation team will drop everything and immediately service your documentation request, but I do guarantee that your request will go into the queue of things we will eventually tackle. https://docs.ceph.com Zac Dover Upstream Documentation Ceph Foundation

1 0

Re: Debian update to 16.2.11-1~bpo11+1 failing
by Matthias Aebi 31 Jan '23

31 Jan '23

I can only confirm that. The file https://download.ceph.com/debian-pacific/pool/main/c/ceph/python3-rados_16.… is clearly missing on the ceph download server which makes it impossible to install the upgrade on Debian. And as the previous 16.2.10 Package definition has been replaced with the latest one, this currently makes it impossible to install Pacific on a Debian system.

3 2

Write amplification for CephFS?
by Manuel Holtgrewe 30 Jan '23

30 Jan '23

Dear all, is there information available anywhere about the write amplification for CephFS? I found quite some material on write amplification of VMs using journaled file system on top of RBD but nothing as it relates to CephFS? From my understanding I would expect the following: - for X-rep, data needs to be written to X pgs (factor X) - for k-m EC, data is written to k+m pgs (factor (k+m)/k) Am I correct? How much is added by meta data, WAL? Do I miss anything? Best wishes, Manuel

2 5

rbd online sparsify image
by Jiatong Shen 30 Jan '23

30 Jan '23

Hello community experts, I would like to know the status of rbd image sparsify. From the website, it should be added at Nautilus ( https://docs.ceph.com/en/latest/releases/nautilus/ from pr (26226 <https://github.com/ceph/ceph/pull/26226>,) ) but on mimic it is related again https://docs.ceph.com/en/latest/releases/octopus/) Is it still the same commit? or is there any update/bug fixes that is not backported to 14.x? I am also curious why it is named re-sparsify in the pr title? Thank you very much for the help. -- Best Regards, Jiatong Shen

2 3

OSDs will not start
by Geoffrey Rhodes 30 Jan '23

30 Jan '23

Good day all, I've an issue with a few OSDs (in two different nodes) that attempt to start but fail / crash quite quickly. They are all LVM disks. I've tried upgrading software, health checks on the hardware (nodes and disks) and there doesn't seem to be any issues there. Recently I've had a few "other" disks physically fail in the cluster and now have one PG down which is blocking some IO on CephFS. I've added the output of the osd journalctl and the osd log below in case it's helpful to identify anything obvious. I also set debug bluefs = 20 , saw this in another post. I recently manually upgraded this node to (17.2.0) before the problem began, later to (17.2.5). - The other osds in this node start / run fine. The other node (15.2.17) also has a few osds that will not start and some that run without issue. Could anyone point me in the right direction to investigate and solve my osd issues. https://pastebin.com/3PkCabdf https://pastebin.com/BT9bnhSb Production system mainly used for CephFS OS: Ubuntu 20.04.5 LTS Ceph versions: 15.2.17 - Octopus (one OSD node manually upgraded to 17.2.5 - Quincy) Erasure data pool (K=4, M=2) - The journal's for each osd are co-located on each drive Kind regards Geoffrey Rhodes

1 1