June 2026 - ceph-users - lists.ceph.io

squid 19.2.5 QE validation status
by Yuri Weinstein 08 Jul '26

08 Jul '26

Details of this release are summarized here: https://tracker.ceph.com/issues/76983#note-2 Release Notes - TBD LRC upgrade - N/A Note: please reply to this email with comments/approvals and also add your comments to the tracker above. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Nitzan, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade: Laura, Patrick (see https://tracker.ceph.com/issues/77569, https://github.com/ceph/ceph/pull/69791) upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 1

Re: squid 19.2.5 QE validation status
by Adam Emerson 01 Jul '26

01 Jul '26

On 29/06/2026, Yuri Weinstein via dev wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76983#note-2 > > Release Notes - TBD > LRC upgrade - N/A > > Note: please reply to this email with comments/approvals and also add your > comments to the tracker above. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Nitzan, Afreen, Travis > rgw: Adam Emerson RGW Approved. The one failure is known and due to a mistake in a backported test case.

1 0

An issue during expanding an cluster - one osd getting close to nearfull
by Robert Lukan 30 Jun '26

30 Jun '26

Hi everyone, We are currently in the middle of migrating our infrastructure to a Proxmox/Ceph stack and ran into a concerning edge case while expanding the cluster. I am hoping to get some advice on the safest way to handle this without risking a nearfull OSD. *The Topology & Situation:* *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and osd.41). *Cluster Average:* ~360 PGs per OSD, ~60% utilization. *Ceph Version:* Squid 19.2.3 The last time we were expanding the cluster, we had issues when expanding it with many OSDs at the same time(it was a memory problem on a host), so we decided to be more careful and add 2 disks at the time. During the backfill phase, we noticed osd.40 catching significantly more PGs than osd.41. We paused the migration by setting the nobackfill flag to prevent a potential nearfull situation. *The Current State (ceph osd df):* osd.40: 423 PGs, 72.56% used, osd.41: 307 PGs, 51.87% used, We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we have paused the backfill. If we were to unset nobackfill, the incoming data for those 218 PGs would risk pushing osd.40 past the 85% nearfull threshold. However, our balancer refused to generate any optimization plans. It reports: "optimize_result": "Unable to find further optimization... or distribution is already perfect" We "understand" why this is happening: the balancer is evaluating only when backfilling is done. However, it is completely ignoring the transient acting state, which was heavily overloading osd.40 *Our Temporary Workaround:* To prevent the potential issue, we have temporarily set the reweight of osd.40 to 0 (effectively draining it out) so we can safely lift the nobackfill flag and let the cluster settle without hitting capacity limits. *My Questions for the Community:* What is the standard best practice for handling this "small bucket" transient imbalance when adding a small number of OSDs to a new host? Or is this just a bad idea in general ? Is there a way to force the built-in upmap balancer to respect the transient/acting capacity of an OSD so it doesn't allow a drive to go towards 85% during a migration? I understand we were not there, but being 12 % away was "close" enough. One of the ideas was to reweight the offending OSD, to remove a few PGs, so it would not be so close to nearfull situation. But this approach is tedious and requires "baby-sitting". I guess this might be a good candidate for upmap remap script, but we were not "brave" enough to run in it a production environment the first time. Thanks, Robert

1 0

Re: squid 19.2.5 QE validation status
by Travis Nielsen 30 Jun '26

30 Jun '26

Rook tests against squid devel is passing, so no concerns from my perspective. Travis On Mon, Jun 29, 2026 at 8:29 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76983#note-2 > > Release Notes - TBD > LRC upgrade - N/A > > Note: please reply to this email with comments/approvals and also add your > comments to the tracker above. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Nitzan, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade: Laura, Patrick (see https://tracker.ceph.com/issues/77569, > https://github.com/ceph/ceph/pull/69791) > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: "cephadm shell" container missing permissions?
by mailing-lists 30 Jun '26

30 Jun '26

Thanks, that worked flawlessly! On 29.06.26 18:17, Eugen Block via ceph-users wrote: > You need to specify the daemon: > > cephadm shell --name osd.X > > Then the device should be visible within the shell. > And please don’t use ceph-volume on an orchestrated cluster outside of > the shell. This will most likely cause issues. > > Zitat von mailing-lists via ceph-users <ceph-users(a)ceph.io>: > >> Hey Cephers, >> >> The big picture is, that I am trying to migrate a DB to a different >> DB-Device. I used to do this many times on Ubuntu 22.04 and >> Reef. With Ubuntu 24.04 and Squid however, I am facing a "permission >> denied" scenario, that I do not understand. It seems like the >> "cephadm shell" container is missing permissions to interact with >> /dev/ceph-* lvs. >> >> Our setup is containerized with Docker and deployed with cephadm. >> >> >> From within "cephadm shell": >> >> "ls -alh >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f065 >> >> ls: cannot access >> '/dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650': >> No such file or directory" >> >> From root on the same host: >> "ls -alh >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 >> lrwxrwxrwx 1 root root 7 Jun 25 17:12 >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 >> -> ../dm-3" >> >> The container just cannot "see" the lv-device. >> >> Something like >> >> "ceph-volume lvm activate 1 91ae2955-577a-4722-8548-5e78372f0650" >> >> will then just fail with: >> ``` >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph >> prime-osd-dir --dev >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 >> --path /var/lib/ceph/osd/ceph-1 --no-mon-config >> stderr: failed to read label for >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650: >> (1) *Operation not permitted* >> stderr: 2026-06-29T15:57:48.848+0000 7e8959bbca80 -1 >> bdev(0x5d2610965800 >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650) >> open stat got: (1) Operation not permitted >> --> RuntimeError: command returned non-zero exit status: 1 >> >> ``` >> >> Has anyone had this problem and solved it? >> >> Is the recommended way to install ceph utilities like ceph-volume on >> the host instead using the toolkit from the cephadm shell? >> >> >> Best >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: "cephadm shell" container missing permissions?
by happy.otter659＠passmail.net 30 Jun '26

30 Jun '26

Hi, I did have this issue but not while migrating a db. Try removing the bluestore tool and any other ceph-* packages that you installed manually, except for cephadm. Then maybe a reatart might be necessary for that node or reatarting the docker daemon. Hope this helps Cheers Best regards - mit freundlichen Gruessen, Christian Frech -------- Original Message -------- On Monday, 06/29/26 at 18:08 mailing-lists via ceph-users <ceph-users(a)ceph.io> wrote: Hey Cephers, The big picture is, that I am trying to migrate a DB to a different DB-Device. I used to do this many times on Ubuntu 22.04 and Reef. With Ubuntu 24.04 and Squid however, I am facing a "permission denied" scenario, that I do not understand. It seems like the "cephadm shell" container is missing permissions to interact with /dev/ceph-* lvs. Our setup is containerized with Docker and deployed with cephadm. From within "cephadm shell": "ls -alh /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f065 0 ls: cannot access '/dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650': No such file or directory" From root on the same host: "ls -alh /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 lrwxrwxrwx 1 root root 7 Jun 25 17:12 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 -> ../dm-3" The container just cannot "see" the lv-device. Something like "ceph-volume lvm activate 1 91ae2955-577a-4722-8548-5e78372f0650" will then just fail with: ``` Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 --path /var/lib/ceph/osd/ceph-1 --no-mon-config stderr: failed to read label for /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650: (1) *Operation not permitted* stderr: 2026-06-29T15:57:48.848+0000 7e8959bbca80 -1 bdev(0x5d2610965800 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650) open stat got: (1) Operation not permitted --> RuntimeError: command returned non-zero exit status: 1 ``` Has anyone had this problem and solved it? Is the recommended way to install ceph utilities like ceph-volume on the host instead using the toolkit from the cephadm shell? Best _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: "cephadm shell" container missing permissions?
by Eugen Block 30 Jun '26

30 Jun '26

You need to specify the daemon: cephadm shell --name osd.X Then the device should be visible within the shell. And please don’t use ceph-volume on an orchestrated cluster outside of the shell. This will most likely cause issues. Zitat von mailing-lists via ceph-users <ceph-users(a)ceph.io>: > Hey Cephers, > > The big picture is, that I am trying to migrate a DB to a different > DB-Device. I used to do this many times on Ubuntu 22.04 and > Reef. With Ubuntu 24.04 and Squid however, I am facing a "permission > denied" scenario, that I do not understand. It seems like the > "cephadm shell" container is missing permissions to interact with > /dev/ceph-* lvs. > > Our setup is containerized with Docker and deployed with cephadm. > > > From within "cephadm shell": > > "ls -alh > /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f065 > > ls: cannot access > '/dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650': No such file or > directory" > > From root on the same host: > "ls -alh > /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 lrwxrwxrwx 1 root root 7 Jun 25 17:12 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 -> > ../dm-3" > > The container just cannot "see" the lv-device. > > Something like > > "ceph-volume lvm activate 1 91ae2955-577a-4722-8548-5e78372f0650" > > will then just fail with: > ``` > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir --dev > /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 --path /var/lib/ceph/osd/ceph-1 > --no-mon-config > stderr: failed to read label for > /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650: (1) *Operation not > permitted* > stderr: 2026-06-29T15:57:48.848+0000 7e8959bbca80 -1 > bdev(0x5d2610965800 > /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650) open stat got: (1) Operation not > permitted > --> RuntimeError: command returned non-zero exit status: 1 > > ``` > > Has anyone had this problem and solved it? > > Is the recommended way to install ceph utilities like ceph-volume on > the host instead using the toolkit from the cephadm shell? > > > Best > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

"cephadm shell" container missing permissions?
by mailing-lists 30 Jun '26

30 Jun '26

Hey Cephers, The big picture is, that I am trying to migrate a DB to a different DB-Device. I used to do this many times on Ubuntu 22.04 and Reef. With Ubuntu 24.04 and Squid however, I am facing a "permission denied" scenario, that I do not understand. It seems like the "cephadm shell" container is missing permissions to interact with /dev/ceph-* lvs. Our setup is containerized with Docker and deployed with cephadm. From within "cephadm shell": "ls -alh /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f065 0 ls: cannot access '/dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650': No such file or directory" From root on the same host: "ls -alh /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 lrwxrwxrwx 1 root root 7 Jun 25 17:12 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 -> ../dm-3" The container just cannot "see" the lv-device. Something like "ceph-volume lvm activate 1 91ae2955-577a-4722-8548-5e78372f0650" will then just fail with: ``` Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 --path /var/lib/ceph/osd/ceph-1 --no-mon-config stderr: failed to read label for /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650: (1) *Operation not permitted* stderr: 2026-06-29T15:57:48.848+0000 7e8959bbca80 -1 bdev(0x5d2610965800 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650) open stat got: (1) Operation not permitted --> RuntimeError: command returned non-zero exit status: 1 ``` Has anyone had this problem and solved it? Is the recommended way to install ceph utilities like ceph-volume on the host instead using the toolkit from the cephadm shell? Best

1 0

Re: squid 19.2.5 QE validation status
by Ilya Dryomov 29 Jun '26

29 Jun '26

On Mon, Jun 29, 2026 at 4:27 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76983#note-2 > > Release Notes - TBD > LRC upgrade - N/A > > Note: please reply to this email with comments/approvals and also add your comments to the tracker above. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Nitzan, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya rbd and krbd approved. Thanks, Ilya

1 0

Re: mgmt-gateway: hostnames in nginx configuration
by Redouane Kachach 29 Jun '26

29 Jun '26

Hi Robert, The behaviour is different bcz the code is shared with other dashboard usecases where the hostname is needed. Anyway I opened https://tracker.ceph.com/issues/77782 to address the inconsistency. Best, Redouane. On Fri, Jun 26, 2026 at 5:55 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > all upstream configuration sections in nginx.conf of the mgmt-gateway > service use the IP addresses of the hosts in question. > > Except of the dashboard section which uses the hostname. > > Why is that implemented this way? > > We stumbled across this as the hostname resolves to an IP address > different from the Ceph public network. This means port 8443 is not > reachable via this IP. > > If the dashboard section would also use the IP address from the > orchestrator's host list this would work. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

CEPHADM_FAILED_DAEMON: OSDs in error state but containers running fine (19.2.3/Squid)
by Sroush Pasandideh 29 Jun '26

29 Jun '26

Hi all, Ceph 19.2.3 (Squid), cephadm-deployed, 528 OSDs. After a drive failure on one host (sdc, ~10 min of SCSI retry stalls at boot), ceph health detail now reports 13 CEPHADM_FAILED_DAEMON warnings on that host: daemon osd.266 on ceph-bh03 is in error state daemon osd.267 on ceph-bh03 is in error state ... (13 total) ceph orch ps confirms the same 13 as error. But checking each container directly: podman ps -a --filter name=osd.275 ... Up 3 days 12 of the 13 containers are running fine, no restarts, clean RocksDB stats in their logs. Only 1 (the OSD that actually sat on the dead drive) has no container at all. So it looks like cephadm's daemon-state cache went stale for the 12 healthy ones. Has anyone seen this before? Is "ceph mgr fail" the right way to force a clean orchestrator re-sync here, or is there a more targeted fix (per-daemon redeploy, restarting the cephadm mgr module)? Thanks, Soroush -- Soroush Pasandideh | Universität Osnabrück - Rechenzentrum | 0541 969 2336

1 0

Re: Deploying newer Ganesha version with older Ceph release
by Nigel Williams 29 Jun '26

29 Jun '26

On Sun, 28 Jun 2026 at 23:39, Arcartis via ceph-users <ceph-users(a)ceph.io> wrote: several good questions about Ganesha versions etc. Thank you for asking these questions, we have been wondering about the same things.

1 0

Deep scrub throughput significantly decreased after upgrade from 19.2.3 to 20.2.1
by Kamil Madac 29 Jun '26

29 Jun '26

Hi, after upgrading one of our production clusters from Ceph 19.2.3 to 20.2.1 we observed that deep scrubbing is no longer keeping up with the configured schedule. The cluster is healthy apart from: HEALTH_WARN 400 pgs not deep-scrubbed in time The number of overdue PGs is steadily increasing (e.g. 0 -> 400 within 4 days). Cluster Ceph 20.2.1 (upgraded from 19.2.3 approximately one week ago) 18 OSDs NVMe SSDs 2208 PGs ~27 TiB user data ~64% raw utilization No recovery/backfill All PGs are active+clean 18 osds: 18 up, 18 in flags: noautoscale OSD latencies are essentially zero: ceph osd perf commit_latency = 0 ms apply_latency = 0 ms Scrub configuration The scrub window has been unchanged for a long time: osd_scrub_begin_hour = 21 osd_scrub_end_hour = 6 The cluster kept up with this configuration before the upgrade. The scheduler configuration is: osd_op_queue = mclock_scheduler osd_mclock_profile = high_recovery_ops This profile was already configured before the upgrade. Observation: I collect the ceph_pg_scrubbing Prometheus metric. Before the upgrade (19.2.3): scrubbing starts around 23:00 continues throughout the whole scrub window there are continuously 1-5 PGs being scrubbed during the night After the upgrade (20.2.1): there is a large initial burst (around 30 PGs scrubbing simultaneously) after that, scrubbing almost completely stops only occasional individual PGs are scrubbed during the rest of the scrub window As a result, the number of overdue deep scrubs increases every day. Questions: Were there any changes to scrub scheduling or mClock behavior between 19.2.3 and 20.2.1 that could explain this? Is this a known issue in 20.2.1? Is high_recovery_ops known to affect deep scrub scheduling differently in Ceph 20 compared to Ceph 19? Are there any debug commands or perf counters that would help determine why new scrub jobs are not being scheduled after the initial burst? I'd appreciate any suggestions on what to investigate next. Thanks! Kamil Madac

1 0

Re: Tentacle 20.2.1 - All MGRs failed
by Anthony D'Atri 28 Jun '26

28 Jun '26

Something else to try might be to delete the .mgr pool and let it be recreated. > On Jun 28, 2026, at 9:31 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I don’t have a response to all your questions, but will try to answer a few. > > Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > >> Due to a number of power and heat events, I had to power down a small ceph cluster last night. There was only time to do a operating system shutdown of all nodes. >> >> When the systems rebooted a few hours later, all was ok except that all 3 MGRs would abort and restart until they reached the restart limit. This left the system with no MGRs and no orchestration. All OSD, MDS, RGW & monitoring subsystems were working as normally as could be expected without an MGR. >> >> Analysis of the log files showed each MGR aborting with the assertion below. Every one was in the telemetry thread. I disabled telemetry and the MGRs could be restarted normally. Questions: >> >> * What caused the assertion error? Was it related to the total shutdown? >> * Aborting the entire MGR seems overkill. Disabling the telemetry >> module and raising a warning seems more proportionate. >> * When running with no active MGR the client IO stats in ceph -s were >> frozen (at plausible non-zero values). Is that expected with no >> active MGR? > > Yes, OSDs report their stats to the MGR which is responsible for displaying those in tje status. > >> * With no active MGR a large number of deep-scrubs started - roughly 3 >> times the maximum number we ever see were actively deep-scrubbing, >> with none in the scrub wait state, and response times were horrible. >> Is this expected with no MGR to control it properly? I had to >> temporarily disable scrubs and deep-scrubs as a workaround. > > I see the same after restarting an entire cluster (mostly in small lab environments), I don’t think that has to do with no active mgr. Although mgr and osd have each a deep-scrub interval, the OSDs are responsible for scheduling. So this is most likely just how OSDs work after a cluster reboot. > >> I haven't yet tried 20.2.2, but I couldn't see anything obvious in the release notes. (Although there were about 3 PRs that I thought might have had the potential to upset exception handling...). >> >> Thanks, Chris >> >> >> { >> "crash_id": "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", >> "timestamp": "2026-06-26T01:46:00.265207Z", >> "process_name": "ceph-mgr", >> "entity_name": "mgr.ceph2", >> "ceph_version": "20.2.1", >> "utsname_hostname": "ceph2", >> "utsname_sysname": "Linux", >> "utsname_release": "7.0.12-201.fc44.x86_64", >> "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", >> "utsname_machine": "x86_64", >> "os_name": "CentOS Stream", >> "os_id": "centos", >> "os_version_id": "9", >> "os_version": "9", >> "assert_condition": "cursor != root", >> "assert_func": "virtual void PyFormatter::close_section()", >> "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", >> "assert_line": 84, >> "assert_thread_name": "telemetry", >> "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread 7f752186e640 time 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", >> "backtrace": [ >> "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", >> "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", >> "raise()", >> "abort()", >> "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f75aa9ed95b]", >> "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", >> "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", >> "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", >> "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", >> "_PyFunction_Vectorcall()", >> "_PyEval_EvalFrameDefault()", >> "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", >> "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", >> "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", >> "_PyObject_CallMethod_SizeT()", >> "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", >> "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", >> "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" >> ] >> } >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Deploying newer Ganesha version with older Ceph release
by Arcartis 28 Jun '26

28 Jun '26

Hi all, can someone tell me why Ceph Squid comes with such an old Ganesha version (5.9)? We are seeing issues with File Locks that cannot be explained, and that I would like to resolve. It does not seem individual components can be upgraded without upgrading the whole cluster. Red Hat has recently commited a code change (https://github.com/nfs-ganesha/nfs-ganesha/commit/baa956ab280ba5a0c42219efd…) that seems like it could be related and mitigate this. But only for Ganesha V10 and not yet in the stable Ceph release if I am not mistaken. I would like to know what is considered stable for Ganesha in relation to the Ceph releases. Has anybody information on this? A compatibility matrix would be great, perhaps I missed something. I want to make sure that e.g. Ganesha V10 isn't causing issues with Ceph Squid or Tentacle FSAL backends. It is on the table to decomission the cephadm NFS daemons and deploy a standalone Ganesha version 10 on the Ceph nodes outside of the orchestrator. I have my reservations that this is a good idea for production. Any guidance is appreciated! Thanks

1 0

Re: Tentacle 20.2.1 - All MGRs failed
by Eugen Block 28 Jun '26

28 Jun '26

Hi, I don’t have a response to all your questions, but will try to answer a few. Zitat von Chris Palmer via ceph-users <ceph-users(a)ceph.io>: > Due to a number of power and heat events, I had to power down a > small ceph cluster last night. There was only time to do a operating > system shutdown of all nodes. > > When the systems rebooted a few hours later, all was ok except that > all 3 MGRs would abort and restart until they reached the restart > limit. This left the system with no MGRs and no orchestration. All > OSD, MDS, RGW & monitoring subsystems were working as normally as > could be expected without an MGR. > > Analysis of the log files showed each MGR aborting with the > assertion below. Every one was in the telemetry thread. I disabled > telemetry and the MGRs could be restarted normally. Questions: > > * What caused the assertion error? Was it related to the total shutdown? > * Aborting the entire MGR seems overkill. Disabling the telemetry > module and raising a warning seems more proportionate. > * When running with no active MGR the client IO stats in ceph -s were > frozen (at plausible non-zero values). Is that expected with no > active MGR? Yes, OSDs report their stats to the MGR which is responsible for displaying those in tje status. > * With no active MGR a large number of deep-scrubs started - roughly 3 > times the maximum number we ever see were actively deep-scrubbing, > with none in the scrub wait state, and response times were horrible. > Is this expected with no MGR to control it properly? I had to > temporarily disable scrubs and deep-scrubs as a workaround. I see the same after restarting an entire cluster (mostly in small lab environments), I don’t think that has to do with no active mgr. Although mgr and osd have each a deep-scrub interval, the OSDs are responsible for scheduling. So this is most likely just how OSDs work after a cluster reboot. > I haven't yet tried 20.2.2, but I couldn't see anything obvious in > the release notes. (Although there were about 3 PRs that I thought > might have had the potential to upset exception handling...). > > Thanks, Chris > > > { > "crash_id": > "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", > "timestamp": "2026-06-26T01:46:00.265207Z", > "process_name": "ceph-mgr", > "entity_name": "mgr.ceph2", > "ceph_version": "20.2.1", > "utsname_hostname": "ceph2", > "utsname_sysname": "Linux", > "utsname_release": "7.0.12-201.fc44.x86_64", > "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", > "utsname_machine": "x86_64", > "os_name": "CentOS Stream", > "os_id": "centos", > "os_version_id": "9", > "os_version": "9", > "assert_condition": "cursor != root", > "assert_func": "virtual void PyFormatter::close_section()", > "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", > "assert_line": 84, > "assert_thread_name": "telemetry", > "assert_msg": > "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function > 'virtual void PyFormatter::close_section()' thread 7f752186e640 time > 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != > root)\n", > "backtrace": [ > "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", > "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", > "raise()", > "abort()", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x18e) [0x7f75aa9ed95b]", > "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", > "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, > std::char_traits<char>, std::allocator<char> > const&, > std::__cxx11::basic_string<char, std::char_traits<char>, > std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", > "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", > "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", > "_PyFunction_Vectorcall()", > "_PyEval_EvalFrameDefault()", > "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", > "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", > "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", > "_PyObject_CallMethod_SizeT()", > "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", > "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", > "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" > ] > } > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Unknown daemon type ceph-exporter
by John Mulligan 28 Jun '26

28 Jun '26

On Thursday, June 25, 2026 9:22:58 AM Eastern Daylight Time Eugen Block via ceph-users wrote: > Hi, > > is this a known issue that ceph-exporter is not recognized by cephadm? > > host1:~ # cephadm --image ceph:v20.2.2 ls | jq -r '.[] | > select(.name=="ceph-exporter.host1") | "\(.name) \(.version)"' > version for unknown daemon type ceph-exporter > ceph-exporter.host1 null > > > I don't see it in the daemon_types listed here: > > https://github.com/ceph/ceph/blob/main/src/cephadm/cephadmlib/listing_update > rs.py > The issue is that a lot of things in cephadm are implemented as a sequence of if statements (per service or service type) and as opposed to using common interfaces and so if something is forgotten or overlooked in the sequence of ifs it fails. Other parts of cephadm should handle ceph-exporter fine, it's just that this version gathering function doesn't handle it properly. It's a bug, please file a tracker. > And one more thing: why is the version empty in 'cephadm ls' output > while 'ceph orch ps' is capable of recognizing the version: > > host1:~ # ceph orch ps --daemon-type ceph-exporter -f json | jq -r > '.[] | select(.daemon_name=="ceph-exporter.host1") | "\(.version)"' > 20.2.2 > I'm not sure about this but it's clearly part of the bug where the system is following different code paths.

2 2

Recovery using OSDs after total mon store loss
by Eugen Block 28 Jun '26

28 Jun '26

Hi, although there is already a section in the upstream docs about mon store recovery using OSDs [0], I decided to write a blog post [1] with more details especially wrt cephadm managed clusters. I tested the procedure multiple times on two different lab clusters and it worked every time. Although I hope that nobody will have to use it (I doubt it though), I do hope that this will help speed up recovery. Maybe the upstream docs could be extended a bit as well. Although I have tested it extensively I may have forgotten something or made a mistake somewhere. I'd appreciate comments if you find anything like that. Have a great rest of the weekend! Eugen [0] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… [1] https://heiterbiswolkig.blogs.nde.ag/2026/06/28/cephadm-recover-mon-store-u…

1 0

mgmt-gateway: hostnames in nginx configuration
by Robert Sander 26 Jun '26

26 Jun '26

Hi, all upstream configuration sections in nginx.conf of the mgmt-gateway service use the IP addresses of the hosts in question. Except of the dashboard section which uses the hostname. Why is that implemented this way? We stumbled across this as the hostname resolves to an IP address different from the Ceph public network. This means port 8443 is not reachable via this IP. If the dashboard section would also use the IP address from the orchestrator's host list this would work. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Tentacle 20.2.1 - All MGRs failed
by Chris Palmer 26 Jun '26

26 Jun '26

Due to a number of power and heat events, I had to power down a small ceph cluster last night. There was only time to do a operating system shutdown of all nodes. When the systems rebooted a few hours later, all was ok except that all 3 MGRs would abort and restart until they reached the restart limit. This left the system with no MGRs and no orchestration. All OSD, MDS, RGW & monitoring subsystems were working as normally as could be expected without an MGR. Analysis of the log files showed each MGR aborting with the assertion below. Every one was in the telemetry thread. I disabled telemetry and the MGRs could be restarted normally. Questions: * What caused the assertion error? Was it related to the total shutdown? * Aborting the entire MGR seems overkill. Disabling the telemetry module and raising a warning seems more proportionate. * When running with no active MGR the client IO stats in ceph -s were frozen (at plausible non-zero values). Is that expected with no active MGR? * With no active MGR a large number of deep-scrubs started - roughly 3 times the maximum number we ever see were actively deep-scrubbing, with none in the scrub wait state, and response times were horrible. Is this expected with no MGR to control it properly? I had to temporarily disable scrubs and deep-scrubs as a workaround. I haven't yet tried 20.2.2, but I couldn't see anything obvious in the release notes. (Although there were about 3 PRs that I thought might have had the potential to upset exception handling...). Thanks, Chris { "crash_id": "2026-06-26T01:46:00.265207Z_b0579062-d65c-4a51-8dbc-d9837209aa80", "timestamp": "2026-06-26T01:46:00.265207Z", "process_name": "ceph-mgr", "entity_name": "mgr.ceph2", "ceph_version": "20.2.1", "utsname_hostname": "ceph2", "utsname_sysname": "Linux", "utsname_release": "7.0.12-201.fc44.x86_64", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 01:30:16 UTC 2026", "utsname_machine": "x86_64", "os_name": "CentOS Stream", "os_id": "centos", "os_version_id": "9", "os_version": "9", "assert_condition": "cursor != root", "assert_func": "virtual void PyFormatter::close_section()", "assert_file": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h", "assert_line": 84, "assert_thread_name": "telemetry", "assert_msg": "/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: In function 'virtual void PyFormatter::close_section()' thread 7f752186e640 time 2026-06-26T01:46:00.263380+0000\n/ceph/rpmbuild/BUILD/ceph-20.2.1/src/mgr/PyFormatter.h: 84: FAILED ceph_assert(cursor != root)\n", "backtrace": [ "/lib64/libc.so.6(+0x3fc30) [0x7f75a9eb8c30]", "/lib64/libc.so.6(+0x8d05c) [0x7f75a9f0605c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7f75aa9ed95b]", "/usr/bin/ceph-mgr(+0x6f7a5) [0x55ac462367a5]", "(ActivePyModules::get_perf_schema_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x15a3) [0x55ac462d3143]", "/usr/bin/ceph-mgr(+0x12ba12) [0x55ac462f2a12]", "/lib64/libpython3.9.so.1.0(+0x15b4b7) [0x7f75ab5e34b7]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x12ec69) [0x7f75ab5b6c69]", "_PyFunction_Vectorcall()", "_PyEval_EvalFrameDefault()", "/lib64/libpython3.9.so.1.0(+0x14169b) [0x7f75ab5c969b]", "/lib64/libpython3.9.so.1.0(+0x14fa85) [0x7f75ab5d7a85]", "/lib64/libpython3.9.so.1.0(+0x138725) [0x7f75ab5c0725]", "_PyObject_CallMethod_SizeT()", "(PyModuleRunner::serve()+0x66) [0x55ac463759e6]", "(PyModuleRunner::PyModuleRunnerThread::entry()+0x139) [0x55ac463761e9]", "/lib64/libc.so.6(+0x8b31a) [0x7f75a9f0431a]" ] }

1 0

Q2 2026 Ceph Foundation Newsletter
by Anthony Middleton 26 Jun '26

26 Jun '26

Hey Ceph Friends, Here is the Q2 2026 Ceph Foundation newsletter. During the second quarter, the Ceph Foundation focused on strengthening collaboration across the project, expanding community programs, supporting documentation efforts, and investing in regional events. This edition highlights recent governance discussions, community elections, new contributor resources, Ceph Days events, and opportunities for community members to get involved. If you have an idea for a Ceph-related event, outreach effort, or project, we encourage you to submit a funding request <https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717> and work with the Foundation to help bring it to life. In This Issue - Board and CSC Alignment on Community Priorities - 2026 CSC Election Recap - Meet the New Ceph Technical Writer - Ceph Days Continue to Grow - Latest Ceph Releases - Community Committees and New Ways to Contribute Board and CSC Alignment on Community Priorities The Ceph Governing Board and Community Steering Committee (CSC) met this quarter to discuss the long-term health of the project, governance, documentation, community growth, and Foundation-supported initiatives. The discussion reaffirmed the distinct roles of each organization within the Ceph ecosystem. The CSC continues to provide technical governance for the project, while the Ceph Foundation focuses on community growth, events, outreach, and funding. The Linux Foundation provides operational, legal, and financial support that enables both groups to operate effectively. A significant area of focus was improving coordination between technical priorities and Foundation-supported programs. Current documentation initiatives include improvements to the documentation backport process, development of a new OSD operations guide, clarification of package and container documentation, and the removal of outdated content. The Board and CSC also discussed the future of community events and the importance of establishing clear goals and success metrics for Foundation-funded programs. Both groups reaffirmed their commitment to supporting collaboration between users, developers, operators, and organizations across the Ceph ecosystem. Key Takeaways - Continued alignment between the Board and CSC on project priorities. - Improved coordination around documentation and community outreach efforts. - Ongoing work to define success metrics for Foundation-funded programs and events. - Shared commitment to strengthening collaboration across the community. 2026 CSC Election Recap This year's CSC election introduced several improvements, including ranked-choice voting and candidate statements, helping create a more transparent and participatory election process. The election also highlighted opportunities to improve ballot communication and voter outreach in future cycles. Lessons learned from this year's process will help strengthen future elections and improve the overall voter experience. 2026 CSC Co-Chairs - Patrick Donnelly - Neha Ojha - Dan van der Ster Thank you to all candidates who participated and to every community member who cast a ballot. Read the full election recap for additional details. <https://pad.ceph.com/p/CSC_Election_April_2026> Meet the New Ceph Technical Writer Earlier this year, the Foundation announced its search for a Technical Writer to help improve the quality and accessibility of Ceph documentation. Following the interview process, the hiring committee selected Emmanuel Ameh for the role. Emmanuel will work closely with contributors across the community to improve documentation, reduce onboarding friction for new users, and help make Ceph documentation easier to navigate and maintain. If you are interested in joining the Ceph Tech Writing committee, please contact Emmanuel at eameh(a)contractor.linuxfoundation.org. Read Emmanuel's introduction on the Ceph blog <https://ceph.io/en/news/blog/2026/from-laminated-pages-to-petabytes/>to learn more about his background and goals for the role. Ceph Days Continue to Grow Community events remain one of the most effective ways for Ceph users and contributors to connect, share experiences, and learn from one another. During the second quarter, the community gathered for two Ceph Days events: Ceph Days Seattle 2026 Date: May 28, 2026 Sponsors: OSNexus and CLYSO Attendance: 30 community members John Shubeck, a Senior Storage Technical Specialist for IBM Object Storage, shared a detailed recap of the recent Ceph Days event in a new blog post <https://ceph.io/en/news/blog/2026/cd-seattle-cloud-blog/>. Ceph Days London 2026 Date: June 17, 2026 Lead Sponsor: IBM Attendance: 120 Here are a few of the photos of the thriving Ceph community in London <https://drive.google.com/drive/folders/1qKCSZyephZp42BBqaxw-mQ1iuGnUW6sO?us…> ! These events featured technical presentations, real-world deployment stories, community networking, and discussions about the future of Ceph. The Foundation continues to focus on supporting regional events, meetups, and community-led initiatives that help expand Ceph's reach and introduce new users to the project. Check the upcoming and tentative Ceph Days events for the rest of 2026. Ceph, OpenInfra, and KCD Days Korea - Lead Sponsors: Cloud Native Computing Foundation and OpenInfra - Date: Q3 - URL: TBD Ceph Days Zurich 2026 - Lead Sponsor: CLYSO - Date: Oct 14, 2026 - URL: https://ceph.io/en/community/events/2026/ceph-days-zurich/ Ceph Days New York - Lead Sponsor: Bloombreg - Date: Q3 - URL: TBD Ceph Days Chicago - Lead Sponsor: CLYSO - Date: Q4 - URL: TBD Get Involved Interested in organizing a meetup, hosting a Ceph Day, speaking at an event, or promoting Ceph in your region? Community participation is what makes these events successful. If you need support for a Ceph event, meetup, travel, or community campaign, submit a funding request <https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717>to the Ceph Foundation. Requests are reviewed by the Governing Board and Linux Foundation staff and considered based on available budget and community impact. Latest Ceph Releases The community published maintenance releases for both the Squid and Tentacle release series during the quarter. - v19.2.4 Squid Released <https://ceph.io/en/news/blog/2026/v19-2-4-squid-released/> - v20.2.1 Tentacle Released <https://ceph.io/en/news/blog/2026/v20-2-1-tentacle-released/> - v20.2.2 Tentacle Released <https://ceph.io/en/news/blog/2026/v20-2-2-tentacle-released/> Thank you to the many contributors, maintainers, testers, and reviewers who continue to support Ceph releases. Community Committees: More Ways to Contribute Community committees are coming soon. These committee will provide opportunities for contributors to help shape the future of Ceph beyond code contributions alone. Whether your interests include events, documentation, outreach, onboarding, or community programs, there are many ways to participate. Committee participation will be shared through our social channel and mailing list <https://ceph.io/en/community/connect/>. Watch upcoming community communications for additional opportunities to join working groups and contribute to Foundation-led initiatives. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Unknown daemon type ceph-exporter
by Eugen Block 25 Jun '26

25 Jun '26

Hi, is this a known issue that ceph-exporter is not recognized by cephadm? host1:~ # cephadm --image ceph:v20.2.2 ls | jq -r '.[] | select(.name=="ceph-exporter.host1") | "\(.name) \(.version)"' version for unknown daemon type ceph-exporter ceph-exporter.host1 null I don't see it in the daemon_types listed here: https://github.com/ceph/ceph/blob/main/src/cephadm/cephadmlib/listing_updat… And one more thing: why is the version empty in 'cephadm ls' output while 'ceph orch ps' is capable of recognizing the version: host1:~ # ceph orch ps --daemon-type ceph-exporter -f json | jq -r '.[] | select(.daemon_name=="ceph-exporter.host1") | "\(.version)"' 20.2.2 Thanks! Eugen

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Alex Gorbachev 25 Jun '26

25 Jun '26

> > since I just stumbled across 'ceph cephadm config-check', is it > enabled in this cluster? Could the MGR have warned you with the health > check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before > deploying OSDs? It seems that it's not enabled by default. This plausibly could have warned after hosts were added, before OSD deployment, assuming the hosts had already been added to cephadm and the check was enabled. And one more question: could 'osd_crush_initial_weight = 0' have > prevented this from happening? We have that set in our own cluster to > carefully check if all OSDs are deployed correctly before reweighting. I doubt this would have helped because the CRUSH weight just controls data placement, but osd heartbeat comes at a lower level than that, and that would have fired anyway. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 5:50 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > since I just stumbled across 'ceph cephadm config-check', is it > enabled in this cluster? Could the MGR have warned you with the health > check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before > deploying OSDs? > > It seems that it's not enabled by default. > > And one more question: could 'osd_crush_initial_weight = 0' have > prevented this from happening? We have that set in our own cluster to > carefully check if all OSDs are deployed correctly before reweighting. > > Thanks, > Eugen > > Zitat von Alex Gorbachev via ceph-users <ceph-users(a)ceph.io>: > > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of > peers > > an OSD must be able to reach before it can accurately mark itself up or > > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > > heartbeat decisions are made independently of monitor quorum health. An > OSD > > can begin reporting peers as down based on heartbeat failures even when > > monitor quorum remains healthy. > > I was adding a second rack of OSD servers to an existing installation, > > assuming that the worst that could happen is that those will be > > unreachable. I also missed setting up the cluster_network VLAN on one > side > > (in the 4x100GbE bond). > > > > I added all the new OSDs at once. The moment that happened, the existing > > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > > > The take away in my recommendation is to add OSDs in such a way that > > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time > to > > fix connectivity issues before dealing with down OSDs, as I experienced. > > -- > > Alex Gorbachev > > ISS/Storcium > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Announcing go-ceph v0.40.0
by John Mulligan via dev 25 Jun '26

25 Jun '26

We're happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.40.0 Enhancements include additions to the smb admin package, and a new command builder package. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. PS. The actual release was created June 16th but we did not announce it on the mailing list until now. Apologies for the delayed announcement.

1 0

Re: OSD don't ant to start
by Gilles Mocellin 25 Jun '26

25 Jun '26

Hi ! Of course, I reset thoses settings right after starting the OSD. When I restart 20 times, it was with the default values (40). I only tried once with 1000, and as it didn't worked, I posted my first mail here... If I had tried a second time, it would have probably worked. Le 24 juin 2026 08:42:21 GMT+02:00, Eugen Block via ceph-users <ceph- users(a)ceph.io> a écrit : >Although I wouldn't have expected it to take 20 restarts, it's still good that it helped resolving the issue. I'm not entirely sure of the implications having increased osd_map_share_max_epochs that much. Did you set it back to the default? If not, I recommend to monitor disk usage closely (monitor store), and maybe OSD memory usage as well. > >Zitat von Gilles Mocellin via ceph-users <ceph-users(a)ceph.io>: > >> Le mardi 23 juin 2026, 11:18:45 heure d’été d’Europe centrale Eugen Block via >> ceph-users a écrit : >>> Hi, >>> >>> this sounds familiar: >>> >>> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ Y2YKHQZSBYA7 >>> GS4D4YSPSNMWNE5QMETH/#ZTUQ2CE5547LOPBGQOXHUF63F6UU3RKO >>> >>> You could set the OSD "in" and then retry starting it a couple of times. >>> >>> Regards, >>> Eugen >> >> Yes that sound really that. >> There's a RedHat solution about that (behind a closed wall), they suggest to >> set to 1000 thoses settings : >> osd_map_share_max_epochs and another (I can't access it from here...). >> >> It didn't work at first so I thought it wasn't it. >> But I certainly was far behind more than 1000... >> My OSD stayed down/out for at least 4 days. >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Alex Gorbachev 24 Jun '26

24 Jun '26

Hi Wannes, The 33% threshold is not applied against the raw total number of OSDs in the cluster. It is applied against the OSD’s heartbeat peer set. With the cluster network VLAN missing, public-network connectivity to MONs could still work, while OSD heartbeats, replication, recovery, and peering over the cluster network failed. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 1:25 AM Wannes Smet <smet(a)icsense.com> wrote: > Hi Alex, > > Lessons learned are always valuable information to share! Thanks. > > Pretty sure I'm missing out on a subtlety here. I can't really put my > finger on it why only osd_heartbeat_min_healthy_ratio caused this, so some > questions from my side. > > At first glance I'd think 50% is still larger than 33%? I also assume OSDs > can be peers even if they never were able to heartbeat each other before > due to crush. > > Maybe this happened because that peering sets aren't perfectly balanced? > So you might have gotten slightly below that 50%, but still above 33%. > Meaning you're <17% away from 33%, "critical mass". But then some > additional peering/activity or the like happened due to the "shock change", > and the whole effectively dropped below 33% cluster wide due to some > cascading effects? > > Thanks, > > Wannes > ------------------------------ > *From:* Alex Gorbachev via ceph-users <ceph-users(a)ceph.io> > *Sent:* Tuesday, June 23, 2026 18:10 > *To:* ceph-users <ceph-users(a)ceph.io> > *Subject:* [ceph-users] PSA: be aware of osd_heartbeat_min_healthy_ratio > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made independently of monitor quorum health. An OSD > can begin reporting peers as down based on heartbeat failures even when > monitor quorum remains healthy. > I was adding a second rack of OSD servers to an existing installation, > assuming that the worst that could happen is that those will be > unreachable. I also missed setting up the cluster_network VLAN on one side > (in the 4x100GbE bond). > > I added all the new OSDs at once. The moment that happened, the existing > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > The take away in my recommendation is to add OSDs in such a way that > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to > fix connectivity issues before dealing with down OSDs, as I experienced. > -- > Alex Gorbachev > ISS/Storcium > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Anthony D'Atri 24 Jun '26

24 Jun '26

Honestly, this sort of thing is what led me a while back to eschew the replication network whenever possible. I have a cluster today from which I'm going to remove it. > On Jun 24, 2026, at 5:50 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs? > > It seems that it's not enabled by default. > > And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting. I *suspect* that that wouldn't have prevented this particular situation, but it does inform the idea of incremental deployment of OSDs. Back in the day I tripled the size of a Dumpling cluster. It was not pretty.

1 0

Re: PSA: be aware of osd_heartbeat_min_healthy_ratio
by Eugen Block 24 Jun '26

24 Jun '26

Hi, since I just stumbled across 'ceph cephadm config-check', is it enabled in this cluster? Could the MGR have warned you with the health check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before deploying OSDs? It seems that it's not enabled by default. And one more question: could 'osd_crush_initial_weight = 0' have prevented this from happening? We have that set in our own cluster to carefully check if all OSDs are deployed correctly before reweighting. Thanks, Eugen Zitat von Alex Gorbachev via ceph-users <ceph-users(a)ceph.io>: > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made independently of monitor quorum health. An OSD > can begin reporting peers as down based on heartbeat failures even when > monitor quorum remains healthy. > I was adding a second rack of OSD servers to an existing installation, > assuming that the worst that could happen is that those will be > unreachable. I also missed setting up the cluster_network VLAN on one side > (in the 4x100GbE bond). > > I added all the new OSDs at once. The moment that happened, the existing > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > The take away in my recommendation is to add OSDs in such a way that > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to > fix connectivity issues before dealing with down OSDs, as I experienced. > -- > Alex Gorbachev > ISS/Storcium > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph cephadm config-check status
by Eugen Block 24 Jun '26

24 Jun '26

Hi, I noticed that the 'ceph cephadm config-check ls' output doesn't match the 'ceph cephadm config-check status' output, at least it's not what I would have expected. For example, the "STATUS" of all the checks is enabled according to the ls output: ceph cephadm config-check ls NAME HEALTHCHECK STATUS DESCRIPTION kernel_security CEPHADM_CHECK_KERNEL_LSM enabled checks SELINUX/Apparmor profiles are consistent across cluster hosts os_subscription CEPHADM_CHECK_SUBSCRIPTION enabled checks subscription states are consistent for all cluster hosts public_network CEPHADM_CHECK_PUBLIC_MEMBERSHIP enabled check that all hosts have a NIC on the Ceph public_network osd_mtu_size CEPHADM_CHECK_MTU enabled check that OSD hosts share a common MTU setting osd_linkspeed CEPHADM_CHECK_LINKSPEED enabled check that OSD hosts share a common linkspeed network_missing CEPHADM_CHECK_NETWORK_MISSING enabled checks that the cluster/public networks defined exist on the Ceph hosts ceph_release CEPHADM_CHECK_CEPH_RELEASE enabled check for Ceph version consistency - ceph daemons should be on the same release (unless upgrade is active) kernel_version CEPHADM_CHECK_KERNEL_VERSION enabled checks that the MAJ.MIN of the kernel on Ceph hosts is consistent But 'config-check status' shows disabled: ceph cephadm config-check status Disabled I assume that the config-checks would only become active after enabling it: ceph config set mgr mgr/cephadm/config_checks_enabled true ceph cephadm config-check status Enabled Is that assumption correct? I would expect the ls output not to show the STATUS "enabled" unless "config_checks_enabled" is actually enabled. Thanks, Eugen

1 0

Re: Advice needed: CephFS PGs down/unknown after several OSDs failure — implications on marking OSDs lost
by Eugen Block 24 Jun '26

24 Jun '26

Hi, I don't really have simple answers here, but I'll comment inline. Zitat von Alessandro De Salvo via ceph-users <ceph-users(a)ceph.io>: > Dear ceph-users, > > we are looking for an advice before taking any destructive action on > a production Ceph cluster (18.2.4) after a SAN failure. > > We lost access to a Disk System hosting several OSDs. The bulk of > the recovery/backfills did complete, and most PGs are now > `active+clean`, but a small set of PGs remains in status > `unavailable`. The Disk System might still be recoverable, but not > easily or quickly; it may take days, or it may never come back. We > are trying to understand the risk of waiting versus marking the > missing OSDs as lost. > > We have not marked any OSDs as lost yet. > > This is a Ceph Reef cluster, version 18.2.4. > > Sanitized current status: > > health: HEALTH_WARN > 1 filesystem is degraded > 1 MDS reports slow metadata IOs > noscrub,nodeep-scrub flags set > (temporarily enabled by us to reduce scrub load during recovery/rebalancing) > Reduced data availability: 28 PGs inactive, 12 PGs down > slow ops still present > osd: > 86 OSDs total > 69 up/in > 17 OSDs unavailable because of the SAN issue > pgs: > 3320 active+clean > 16 unknown > 11 down > 1 down+remapped > > The problematic PGs are spread across several services/pools: > > Main CephFS data pool: 4 PGs down > Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown > RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown > RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG unknown > RBD/Kubernetes: 1 PG unknown Could you share the following output: ceph osd tree ceph osd pool ls detail ceph health detail > The services we care about most are the filesystems, especially the > main CephFS and the secondary CephFS. > > For the main CephFS, the metadata pool does not appear to have > inactive PGs. The remaining problematic PGs are in the data pool. Of > the main CephFS data PGs, only one appears to contain real data; the > others appear empty. I find it very unlikely that only one of the PGs has data. If there are down OSDs, they can't report to the mgr, so the stats are most likely wrong. > For the non-empty main CephFS data PG, `pg query` shows roughly: > > state: down > acting/up: all current acting OSDs are up > last_update == last_complete > empty: 0 > incomplete: 0 > stat_sum: > num_bytes: ~80 GB > num_objects: ~131k > num_objects_missing: 0 > num_objects_unfound: 0 > num_objects_degraded: 0 > num_objects_misplaced: 0 > blocked: > peering is blocked due to down OSDs > down_osds_we_would_probe: > a few OSDs from previous intervals > peering_blocked_by: > "starting or marking this osd lost may let us proceed" > > The recovery state for this PG includes past intervals with OSDs > that are currently unavailable. The current acting set is up, but > Ceph refuses to activate the PG because it still wants to probe down > OSDs from the PG history. > > We tried to list objects from that PG: > rados -p <main_cephfs_data_pool> ls --pgid <pgid> > > but it hangs/times out, so we cannot currently map RADOS objects > back to CephFS inode/path names. > > For the secondary CephFS, the situation is more concerning because > one metadata PG is `unknown`. Most `pg query` attempts on unknown > PGs return: > Error ENOENT: i don't have pgid <pgid> Same applies here, down OSDs can't report anything. > For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and > down, while some others appear empty. Some metadata PGs are unknown. > RGW can temporarily work after restarting daemons, but eventually > gets stuck again, presumably when requests hit unavailable RADOS > objects or metadata. > > We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD > pool, only one non-critical PVC seems to hang on `rbd info`; most > other images respond. So the main operational concern remains CephFS > and RGW. > > Our questions are: > 1. In a case like this, where a CephFS data PG is `down`, not > `incomplete`, and shows `num_objects_missing = 0` and > `num_objects_unfound = 0`, but peering is blocked by historical down > OSDs, what are the likely consequences of marking those OSDs as lost? Data loss. IIRC, a down OSD means there are no OSDs to query at all (for that PG) while incomplete means that there are not enough OSDs "up" to determine which one has the most up-to-date state. > 2. If we mark the unavailable OSDs as lost, should we expect the > main CephFS to: > - come back with only some files or file extents unreadable/corrupt, > - remain blocked/degraded, > - or risk a wider filesystem-level failure? That's difficult to answer. Losing an entire OSD can mean several TB of data lost, so it's unlikely that "only some files or file extents" will be lost. I think it would allow Ceph to recover further, most likely you'd have to scrub the FS afterwards, but at some point you should have a working FS, I think. > 3. Since the main CephFS metadata pool does not appear to have > inactive PGs, is it reasonable to assume that the risk is mostly > limited to file data objects rather than the entire namespace? Or is > that assumption unsafe? I tend to concur, but what do you mean by "does not appear to have inactive PGs"? The 'ceph health detail' output should have a clear answer to that. > 4. For the secondary CephFS, where a metadata PG is `unknown`, > should we treat declaring OSDs lost as much more dangerous? Could > that make the whole filesystem unusable rather than only losing some > files? There are ways to recover the metadata, but be very careful with that! https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ > 5. Is there any recommended way to map the affected CephFS data PG > to file paths while the PG is still down? `rados ls --pgid` hangs, > so we currently cannot list objects in the PG. I don't think so. > 6. Before declaring the unavailable OSDs lost, are there any > additional diagnostic or recovery steps you would recommend to > better assess the situation or reduce the risk of data loss? Is there no chance to access the missing OSDs? So one could at least export PGs or something? > Any advice would be greatly appreciated. > > Best regards, > > Nadir and Alessandro > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSD don't ant to start
by Eugen Block 24 Jun '26

24 Jun '26

Although I wouldn't have expected it to take 20 restarts, it's still good that it helped resolving the issue. I'm not entirely sure of the implications having increased osd_map_share_max_epochs that much. Did you set it back to the default? If not, I recommend to monitor disk usage closely (monitor store), and maybe OSD memory usage as well. Zitat von Gilles Mocellin via ceph-users <ceph-users(a)ceph.io>: > Le mardi 23 juin 2026, 11:18:45 heure d’été d’Europe centrale Eugen Block via > ceph-users a écrit : >> Hi, >> >> this sounds familiar: >> >> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2YKHQZSBYA7 >> GS4D4YSPSNMWNE5QMETH/#ZTUQ2CE5547LOPBGQOXHUF63F6UU3RKO >> >> You could set the OSD "in" and then retry starting it a couple of times. >> >> Regards, >> Eugen > > Yes that sound really that. > There's a RedHat solution about that (behind a closed wall), they suggest to > set to 1000 thoses settings : > osd_map_share_max_epochs and another (I can't access it from here...). > > It didn't work at first so I thought it wasn't it. > But I certainly was far behind more than 1000... > My OSD stayed down/out for at least 4 days. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: OSD don't ant to start
by Gilles Mocellin 24 Jun '26

24 Jun '26

Le mardi 23 juin 2026, 11:18:45 heure d’été d’Europe centrale Eugen Block via ceph-users a écrit : > Hi, > > this sounds familiar: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2YKHQZSBYA7 > GS4D4YSPSNMWNE5QMETH/#ZTUQ2CE5547LOPBGQOXHUF63F6UU3RKO > > You could set the OSD "in" and then retry starting it a couple of times. > > Regards, > Eugen Yes that sound really that. There's a RedHat solution about that (behind a closed wall), they suggest to set to 1000 thoses settings : osd_map_share_max_epochs and another (I can't access it from here...). It didn't work at first so I thought it wasn't it. But I certainly was far behind more than 1000... My OSD stayed down/out for at least 4 days.

1 0

Re: OSD don't ant to start
by Gilles Mocellin 24 Jun '26

24 Jun '26

Le mardi 23 juin 2026, 10:29:07 heure d’été d’Europe centrale Gilles Mocellin via ceph-users a écrit : > Hello, > > After removing a failing drive with : > ceph orch osd rm ${ID} --zap --replace > > Waiting for recovery (3 days, in a 12 nodes 144 drives)... > > I add the drive with : > ceph orch daemon add osd myserver:/dev/sdn > > In the journal logs of the OSD, I can see : > [...] OSD shutdown > I've found that solution on RedHat site : > https://access.redhat.com/solutions/7029908 > > Which is related to epochs, they have that log before : > 276003 handle_osd_map epochs [276004,276043] i have 276003, src has > [274883, 303268] > 276043 map says i am stopped by admin. shutting down. > > I don't have messages about epochs. > > I tried anyway with thoses settings : > # ceph config set osd osd_map_message_max 1000 > # ceph config set osd osd_map_share_max_epochs 1000 > > But no, the OSD still doesn't start. I try to restart several times again, and see that an ID (seems to be the osdmap epoch) is increasing each time. After something like 20 restarts (systemd doesn't like that, I had to reset- failed the service several times), I reached the last_commited osdmap and the OSD starts !

1 0

PSA: be aware of osd_heartbeat_min_healthy_ratio
by Alex Gorbachev 24 Jun '26

24 Jun '26

osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers an OSD must be able to reach before it can accurately mark itself up or avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD heartbeat decisions are made independently of monitor quorum health. An OSD can begin reporting peers as down based on heartbeat failures even when monitor quorum remains healthy. I was adding a second rack of OSD servers to an existing installation, assuming that the worst that could happen is that those will be unreachable. I also missed setting up the cluster_network VLAN on one side (in the 4x100GbE bond). I added all the new OSDs at once. The moment that happened, the existing OSDs saw 50% unreachable, and all production OSDs promptly shut down. The take away in my recommendation is to add OSDs in such a way that they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to fix connectivity issues before dealing with down OSDs, as I experienced. -- Alex Gorbachev ISS/Storcium

1 0

Re: Ceph Scrub Schedule
by Alex 23 Jun '26

23 Jun '26

Hi Eugen, we do almost the same thing. Funny that we came up with the same approach.

1 0

Re: OSD don't ant to start
by Eugen Block 23 Jun '26

23 Jun '26

Hi, this sounds familiar: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Y2YKHQZSBYA… You could set the OSD "in" and then retry starting it a couple of times. Regards, Eugen Zitat von Gilles Mocellin via ceph-users <ceph-users(a)ceph.io>: > Hello, > > After removing a failing drive with : > ceph orch osd rm ${ID} --zap --replace > > Waiting for recovery (3 days, in a 12 nodes 144 drives)... > > I add the drive with : > ceph orch daemon add osd myserver:/dev/sdn > > In the journal logs of the OSD, I can see : > > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.908+0000 7f1fffa90740 1 osd.66 389707 start_boot > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.908+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_max_backfills set to 1 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_recovery_max_active set to 0 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_recovery_max_active_hdd set to 3 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_options_for_qos osd_recovery_max_active_ssd set to 10 > Jun 23 10:06:51 myserver bash[1944632]: debug > 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 bench > count 12288000 bsize 4 KiB > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.552+0000 7f1fffa90740 1 osd.66 389707 > maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth > (MiB/sec): 6.518 iops: 1668.622 elapsed_sec: 1.798 > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.552+0000 7f1fffa90740 0 log_channel(cluster) > log [WRN] : OSD bench result of 1668.621657 IOPS is not within the > threshold limit range of 50.000000 IOPS and 500.000000 IOPS for > osd.66. IOPS capacity is unchanged at 315.000000 IOPS. The > recommendation is to establish the osd's IOPS capacity using other > benchmark tools (e.g. Fio) and then override > osd_mclock_max_capacity_iops_[hdd|ssd]. > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.552+0000 7f1ffba10640 -1 osd.66 389707 osdmap > says I am destroyed > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1fef00a640 -1 osd.66 389747 map says > i am stopped by admin. shutting down. > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1fef00a640 0 osd.66 389747 > _committed_osd_maps shutdown OSD via async signal > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 received signal: > Interrupt from Kernel ( Could be generated by pthread_kill(), > raise(), abort(), alarm() ) UID: 0 > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 osd.66 389747 *** Got > signal Interrupt *** > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 0 osd.66 389747 Fast > Shutdown: - cct->_conf->osd_fast_shutdown = 1, null-fm = 0 > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 osd.66 389747 *** > Immediate shutdown (osd_fast_shutdown=true) *** > Jun 23 10:06:55 myserver bash[1944632]: debug > 2026-06-23T08:06:55.740+0000 7f1ffca28640 0 osd.66 389747 > prepare_to_stop starting shutdown > Jun 23 10:06:57 myserver systemd[1]: > ceph-02acb271-d31a-4c26-82a2-0e0e1c579ed5(a)osd.66.service: > Deactivated successfully. > > I retry several time, nothing new. > The OSD is mark down and then stops. > > I've found that solution on RedHat site : > https://access.redhat.com/solutions/7029908 > > Which is related to epochs, they have that log before : > 276003 handle_osd_map epochs [276004,276043] i have 276003, src has > [274883, 303268] > 276043 map says i am stopped by admin. shutting down. > > I don't have messages about epochs. > > I tried anyway with thoses settings : > # ceph config set osd osd_map_message_max 1000 > # ceph config set osd osd_map_share_max_epochs 1000 > > But no, the OSD still doesn't start. > > Has anyone a clue ? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: mgmt-gateway - Issue with Prometheus/Alertmanager mTLS
by Redouane Kachach 23 Jun '26

23 Jun '26

Hi Cameron, I'd recommend using the slack channel and posting more info about the kind of errors you are getting. Regards, Redo. On Fri, Jun 19, 2026 at 12:47 PM Cameron Ditchfield via ceph-users < ceph-users(a)ceph.io> wrote: > Hi ! > > Has anyone running Tentacle successfully got the management gateway working > with a non-cephadm-root issued certificate? > > I'm struggling to get it working with Prometheus & Alertmanager when using > a cert not issued by the cephadm-root CA. I have a Tentacle cluster v20.2.1 > that has the management gateway deployed across three nodes with a > keepalived ingress and virtual IP. I've updated the dashboard to point to > the DNS hostname for the VIP for alertmanager and prometheus. > > I can access the dashboard via the hostname, as expected, but there are no > metrics on the overview page and there's a cannot verify SSL cert error for > both alertmanager and prometheus. The CA that issued the gateway cert is > trusted by all nodes in the cluster. > > Thanks, > Cameron > > -- > Thought Machine Group Limited, a company registered in England & Wales. > Registered number: 11114277. > Registered Office: 5 New Street Square, > London EC4A 3TW > < > https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW&entry=gmail… > >. > > > The content of this email is confidential and intended for the recipient > specified in message only. It is strictly forbidden to share any part of > this message with any third party, without a written consent of the > sender. > If you received this message by mistake, please reply to this message and > follow with its deletion, so that we can ensure such a mistake does not > occur in the future. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

OSD don't ant to start
by Gilles Mocellin 23 Jun '26

23 Jun '26

Hello, After removing a failing drive with : ceph orch osd rm ${ID} --zap --replace Waiting for recovery (3 days, in a 12 nodes 144 drives)... I add the drive with : ceph orch daemon add osd myserver:/dev/sdn In the journal logs of the OSD, I can see : Jun 23 10:06:51 myserver bash[1944632]: debug 2026-06-23T08:06:51.908+0000 7f1fffa90740 1 osd.66 389707 start_boot Jun 23 10:06:51 myserver bash[1944632]: debug 2026-06-23T08:06:51.908+0000 7f1fffa90740 1 osd.66 389707 maybe_override_options_for_qos osd_max_backfills set to 1 Jun 23 10:06:51 myserver bash[1944632]: debug 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 maybe_override_options_for_qos osd_recovery_max_active set to 0 Jun 23 10:06:51 myserver bash[1944632]: debug 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 maybe_override_options_for_qos osd_recovery_max_active_hdd set to 3 Jun 23 10:06:51 myserver bash[1944632]: debug 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 maybe_override_options_for_qos osd_recovery_max_active_ssd set to 10 Jun 23 10:06:51 myserver bash[1944632]: debug 2026-06-23T08:06:51.912+0000 7f1fffa90740 1 osd.66 389707 bench count 12288000 bsize 4 KiB Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.552+0000 7f1fffa90740 1 osd.66 389707 maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth (MiB/sec): 6.518 iops: 1668.622 elapsed_sec: 1.798 Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.552+0000 7f1fffa90740 0 log_channel(cluster) log [WRN] : OSD bench result of 1668.621657 IOPS is not within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS for osd.66. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.552+0000 7f1ffba10640 -1 osd.66 389707 osdmap says I am destroyed Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1fef00a640 -1 osd.66 389747 map says i am stopped by admin. shutting down. Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1fef00a640 0 osd.66 389747 _committed_osd_maps shutdown OSD via async signal Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 received signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 osd.66 389747 *** Got signal Interrupt *** Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1ffca28640 0 osd.66 389747 Fast Shutdown: - cct->_conf->osd_fast_shutdown = 1, null-fm = 0 Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1ffca28640 -1 osd.66 389747 *** Immediate shutdown (osd_fast_shutdown=true) *** Jun 23 10:06:55 myserver bash[1944632]: debug 2026-06-23T08:06:55.740+0000 7f1ffca28640 0 osd.66 389747 prepare_to_stop starting shutdown Jun 23 10:06:57 myserver systemd[1]: ceph-02acb271-d31a-4c26-82a2-0e0e1c579ed5(a)osd.66.service: Deactivated successfully. I retry several time, nothing new. The OSD is mark down and then stops. I've found that solution on RedHat site : https://access.redhat.com/solutions/7029908 Which is related to epochs, they have that log before : 276003 handle_osd_map epochs [276004,276043] i have 276003, src has [274883, 303268] 276043 map says i am stopped by admin. shutting down. I don't have messages about epochs. I tried anyway with thoses settings : # ceph config set osd osd_map_message_max 1000 # ceph config set osd osd_map_share_max_epochs 1000 But no, the OSD still doesn't start. Has anyone a clue ?

1 0

Re: Ceph Scrub Schedule
by Eugen Block 23 Jun '26

23 Jun '26

Hi, I'm not aware of any more granular approach. Until a few years ago we used a deep-scrub script via cronjob to configure a ratio we wanted to deep-scrub during a specific time window. During weekends the amount of deep-scrubs was higher than on work days. Back then we had a HDD cluster with a cache-tier on SSD and we needed the performance during working hours. This thread contains some form of it, I think there are several versions of it online: https://forum.proxmox.com/threads/ceph-schedule-deep-scrubs-to-prevent-serv… We upgraded to (almost) all flash in the meantime so we're not that sensitive anymore to the deep-scrub workload. Regards, Eugen Zitat von Alex via ceph-users <ceph-users(a)ceph.io>: > Hello everyone. > > We've been having issues with Deep Scrubs interfering with Ceph-backed VM > performance. > Hardware upgrade is unfortunately out of the question for now. > After much experimenting, asking and searching we've settled on cutting the > scrubs during the weekday hours which is when the deploys happen to a > couple of hours a night and letting the scrubs run for almost the entire > day during the weekend to catch up. There are still missed scrubs by the > EOW but i manually trigger them. It's not great but it works. > > The way we do this is a cron job which changes the schedule on Friday night > and sets it back on Sunday. > > For example > Mon-Fri deep scrub 1am-4am > Sat-Sun deep scrub 1am-11pm > > The first question is there's nothing else i assume we can tweak? The > consecutive scrubs are already set to one, we can't upgrade the hardware, > we tried various scrub windows and just can't fit them in during the > weekdays without the cron job hack. Is there something obvious we missed? > > The second question is, is there really no more granular schedule then > hours during which the deep scrub can run? The cron feels like a dirty hack. > > There's the "osd_scrub_begin/end_week_day" setting but that doesn't do what > we need. > We want to do some scrubs during the week and the rest during the weekend. > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Next Ceph Users & Devs Meetup Wednesday, June 24th, at 7 am PDT
by Anthony Middleton 23 Jun '26

23 Jun '26

Hey Ceph Friends, I hope your week is off to a good start. The next Ceph Users + Devs Meetup is this Wednesday, June 24th, at 7 am PDT. If you'd like to add a topic to the agenda, there's still time. Please add topics to the pad before Tuesday evening so the host has time to review in advance. Ether Pad link - https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have any questions, please let me know. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

CSC Meeting Notes: 2026-06-22
by Adam Emerson 23 Jun '26

23 Jun '26

Ceph Steering Committee Meeting Notes 22 June 2026 * Dan van der Ster: Voting rules discussion/reminder* ** Patrick had kicked off doing Robert's rules of order/Helios ** Last week two motions didn't pass because we didn't have enough votes *** We need to be a bit more tactical. *** We have 34 voting members *** We adopted the technical charter with the voting rules ** There's two thresholds *** We need to have a quorum (50%) we just need a majority to pass a motion (we could have nine people in favor) *** We currently have a quorum, and we're mostly voting members. *** For electronic votes we always need at least 50% of voting members. ** Regarding adding and removing members *** There was no text about that in the new technical charter *** In our governance charter we require a supermajority rule *** The project can come up with its own rule/process for adding/removing members. ** The intuitive/obvious thing is we grandfather in the supermajority rule for adding members *** Last week we had two motions on the floor, one to add Sage McTaggart *** One is to require the security working group, and that should require a majority ** The Helios vote should be out right now *** It should be open now, and should have opened just at the start of this meeting. ** Joseph M: Do we need a vote to create working groups? *** We didn't need a vote to make the Cephalocon working groups, but we decided to have one for the security working group. *** Maybe it's case by case and depends on the function. * Dan van der Ster: Do we want to encourage people to come to this meeting so we get quorum? ** Adam E: Yes. * Neha: Ceph day London update ** IBM and Canonical were the sponsors ** Lots of poeple using Ceph showed up, interesting talks ** New presentations of things landing soon and projects people are building with Ceph. ** It was very successful and this is one geographic area where community events are clearly shining. ** Photos will be uploaded soon. * Cephalocon working group kickoff ** https://tracker.ceph.com/issues/77296 ** Coordinating over Slack * There is a Foundation board meeting tomorrow ** Neha will present feedback from us to the Foundation * The meeting was posed for everyone to vote in Helios so we can get it done * Neja: Anthony Middleton brought it to our notice that we don't have a component lead for NVMEof ** Earlier we had the concept of a component lead being a member of the CSC by default ** Motion that we add Aviv made by Neha and seconded by Yehuda *** Added to a Helios vote

1 0

Ceph Scrub Schedule
by Alex 22 Jun '26

22 Jun '26

Hello everyone. We've been having issues with Deep Scrubs interfering with Ceph-backed VM performance. Hardware upgrade is unfortunately out of the question for now. After much experimenting, asking and searching we've settled on cutting the scrubs during the weekday hours which is when the deploys happen to a couple of hours a night and letting the scrubs run for almost the entire day during the weekend to catch up. There are still missed scrubs by the EOW but i manually trigger them. It's not great but it works. The way we do this is a cron job which changes the schedule on Friday night and sets it back on Sunday. For example Mon-Fri deep scrub 1am-4am Sat-Sun deep scrub 1am-11pm The first question is there's nothing else i assume we can tweak? The consecutive scrubs are already set to one, we can't upgrade the hardware, we tried various scrub windows and just can't fit them in during the weekdays without the cron job hack. Is there something obvious we missed? The second question is, is there really no more granular schedule then hours during which the deep scrub can run? The cron feels like a dirty hack. There's the "osd_scrub_begin/end_week_day" setting but that doesn't do what we need. We want to do some scrubs during the week and the rest during the weekend. Thanks.

1 0

Re: Advice needed: CephFS PGs down/unknown after several OSDs failure — implications on marking OSDs lost
by Sergio Rabellino 22 Jun '26

22 Jun '26

I'm not the best choice for a safe answer, but probably you should also declare the pools replica/erasure configuration and what is the domain failure choice: in my experience losing a node should not bring to pg unknown. Anyway I suggest to first understand why this happened and try to bring all the pg to a clean state, before doing any other action on upper layers. Bests. Il 22/06/2026 12:46, Alessandro De Salvo via ceph-users ha scritto: > > Dear ceph-users, > > we are looking for an advice before taking any destructive action on a > production Ceph cluster (18.2.4) after a SAN failure. > > We lost access to a Disk System hosting several OSDs. The bulk of the > recovery/backfills did complete, and most PGs are now `active+clean`, > but a small set of PGs remains in status `unavailable`. The Disk > System might still be recoverable, but not easily or quickly; it may > take days, or it may never come back. We are trying to understand the > risk of waiting versus marking the missing OSDs as lost. > > We have not marked any OSDs as lost yet. > > This is a Ceph Reef cluster, version 18.2.4. > > Sanitized current status: > > health: HEALTH_WARN > 1 filesystem is degraded > 1 MDS reports slow metadata IOs > noscrub,nodeep-scrub flags set > (temporarily enabled by us to reduce scrub load during > recovery/rebalancing) > Reduced data availability: 28 PGs inactive, 12 PGs down > slow ops still present > osd: > 86 OSDs total > 69 up/in > 17 OSDs unavailable because of the SAN issue > pgs: > 3320 active+clean > 16 unknown > 11 down > 1 down+remapped > > The problematic PGs are spread across several services/pools: > > Main CephFS data pool: 4 PGs down > Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown > RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown > RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG > unknown > RBD/Kubernetes: 1 PG unknown > > The services we care about most are the filesystems, especially the > main CephFS and the secondary CephFS. > > For the main CephFS, the metadata pool does not appear to have > inactive PGs. The remaining problematic PGs are in the data pool. Of > the main CephFS data PGs, only one appears to contain real data; the > others appear empty. > > For the non-empty main CephFS data PG, `pg query` shows roughly: > > state: down > acting/up: all current acting OSDs are up > last_update == last_complete > empty: 0 > incomplete: 0 > stat_sum: > num_bytes: ~80 GB > num_objects: ~131k > num_objects_missing: 0 > num_objects_unfound: 0 > num_objects_degraded: 0 > num_objects_misplaced: 0 > blocked: > peering is blocked due to down OSDs > down_osds_we_would_probe: > a few OSDs from previous intervals > peering_blocked_by: > "starting or marking this osd lost may let us proceed" > > The recovery state for this PG includes past intervals with OSDs that > are currently unavailable. The current acting set is up, but Ceph > refuses to activate the PG because it still wants to probe down OSDs > from the PG history. > > We tried to list objects from that PG: > rados -p <main_cephfs_data_pool> ls --pgid <pgid> > > but it hangs/times out, so we cannot currently map RADOS objects back > to CephFS inode/path names. > > For the secondary CephFS, the situation is more concerning because one > metadata PG is `unknown`. Most `pg query` attempts on unknown PGs return: > Error ENOENT: i don't have pgid <pgid> > > For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and > down, while some others appear empty. Some metadata PGs are unknown. > RGW can temporarily work after restarting daemons, but eventually gets > stuck again, presumably when requests hit unavailable RADOS objects or > metadata. > > We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD > pool, only one non-critical PVC seems to hang on `rbd info`; most > other images respond. So the main operational concern remains CephFS > and RGW. > > Our questions are: > 1. In a case like this, where a CephFS data PG is `down`, not > `incomplete`, and shows `num_objects_missing = 0` and > `num_objects_unfound = 0`, but peering is blocked by historical down > OSDs, what are the likely consequences of marking those OSDs as lost? > > 2. If we mark the unavailable OSDs as lost, should we expect the main > CephFS to: > - come back with only some files or file extents unreadable/corrupt, > - remain blocked/degraded, > - or risk a wider filesystem-level failure? > > 3. Since the main CephFS metadata pool does not appear to have > inactive PGs, is it reasonable to assume that the risk is mostly > limited to file data objects rather than the entire namespace? Or is > that assumption unsafe? > > 4. For the secondary CephFS, where a metadata PG is `unknown`, should > we treat declaring OSDs lost as much more dangerous? Could that make > the whole filesystem unusable rather than only losing some files? > > 5. Is there any recommended way to map the affected CephFS data PG to > file paths while the PG is still down? `rados ls --pgid` hangs, so we > currently cannot list objects in the PG. > > 6. Before declaring the unavailable OSDs lost, are there any > additional diagnostic or recovery steps you would recommend to better > assess the situation or reduce the risk of data loss? > > Any advice would be greatly appreciated. > > Best regards, > > Nadir and Alessandro > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>

1 0

Re: Query regarding Debian Trixie support for Squid and Tentacle
by BASSAGET Cédric 22 Jun '26

22 Jun '26

Hello I think ceph team prefers to maintain quay.io images for containers as they are pushing up cephadm as replacement of legacy clusters. I'm currentrly running an old legacy cluster on debian 12 / ceph 18 and I'm stuck with a python problem (a known pyo3 issue). Trying to move to cephadm but can't load cephadm module because of this pyo3 issue. I'm wayting for a fix / tip / workaround .... Regards Cédric Le ven. 19 juin 2026 à 12:19, Piradeep G via ceph-users <ceph-users(a)ceph.io> a écrit : > Hi everyone, > > I have a question regarding package availability for Debian Trixie. > > Currently, I can see that the Debian-Squid packages are available only for > the Bookworm and Jammy distributions. Are there any plans to provide > Debian-Squid packages for Debian Trixie as well? > > Similarly, I noticed that the Debian-Tentacle packages are currently > available only for Bookworm, Jammy, and Noble. Is support for Debian Trixie > planned for Debian-Tentacle, and if so, is there an estimated timeline for > when it will be available? > > We're evaluating our upgrade plans and would appreciate any information or > insights from the community. > > Thanks in advance! > > Regards, > > Piradeep G > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Advice needed: CephFS PGs down/unknown after several OSDs failure — implications on marking OSDs lost
by Alessandro De Salvo 22 Jun '26

22 Jun '26

Dear ceph-users, we are looking for an advice before taking any destructive action on a production Ceph cluster (18.2.4) after a SAN failure. We lost access to a Disk System hosting several OSDs. The bulk of the recovery/backfills did complete, and most PGs are now `active+clean`, but a small set of PGs remains in status `unavailable`. The Disk System might still be recoverable, but not easily or quickly; it may take days, or it may never come back. We are trying to understand the risk of waiting versus marking the missing OSDs as lost. We have not marked any OSDs as lost yet. This is a Ceph Reef cluster, version 18.2.4. Sanitized current status: health: HEALTH_WARN 1 filesystem is degraded 1 MDS reports slow metadata IOs noscrub,nodeep-scrub flags set (temporarily enabled by us to reduce scrub load during recovery/rebalancing) Reduced data availability: 28 PGs inactive, 12 PGs down slow ops still present osd: 86 OSDs total 69 up/in 17 OSDs unavailable because of the SAN issue pgs: 3320 active+clean 16 unknown 11 down 1 down+remapped The problematic PGs are spread across several services/pools: Main CephFS data pool: 4 PGs down Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG unknown RBD/Kubernetes: 1 PG unknown The services we care about most are the filesystems, especially the main CephFS and the secondary CephFS. For the main CephFS, the metadata pool does not appear to have inactive PGs. The remaining problematic PGs are in the data pool. Of the main CephFS data PGs, only one appears to contain real data; the others appear empty. For the non-empty main CephFS data PG, `pg query` shows roughly: state: down acting/up: all current acting OSDs are up last_update == last_complete empty: 0 incomplete: 0 stat_sum: num_bytes: ~80 GB num_objects: ~131k num_objects_missing: 0 num_objects_unfound: 0 num_objects_degraded: 0 num_objects_misplaced: 0 blocked: peering is blocked due to down OSDs down_osds_we_would_probe: a few OSDs from previous intervals peering_blocked_by: "starting or marking this osd lost may let us proceed" The recovery state for this PG includes past intervals with OSDs that are currently unavailable. The current acting set is up, but Ceph refuses to activate the PG because it still wants to probe down OSDs from the PG history. We tried to list objects from that PG: rados -p <main_cephfs_data_pool> ls --pgid <pgid> but it hangs/times out, so we cannot currently map RADOS objects back to CephFS inode/path names. For the secondary CephFS, the situation is more concerning because one metadata PG is `unknown`. Most `pg query` attempts on unknown PGs return: Error ENOENT: i don't have pgid <pgid> For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and down, while some others appear empty. Some metadata PGs are unknown. RGW can temporarily work after restarting daemons, but eventually gets stuck again, presumably when requests hit unavailable RADOS objects or metadata. We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD pool, only one non-critical PVC seems to hang on `rbd info`; most other images respond. So the main operational concern remains CephFS and RGW. Our questions are: 1. In a case like this, where a CephFS data PG is `down`, not `incomplete`, and shows `num_objects_missing = 0` and `num_objects_unfound = 0`, but peering is blocked by historical down OSDs, what are the likely consequences of marking those OSDs as lost? 2. If we mark the unavailable OSDs as lost, should we expect the main CephFS to: - come back with only some files or file extents unreadable/corrupt, - remain blocked/degraded, - or risk a wider filesystem-level failure? 3. Since the main CephFS metadata pool does not appear to have inactive PGs, is it reasonable to assume that the risk is mostly limited to file data objects rather than the entire namespace? Or is that assumption unsafe? 4. For the secondary CephFS, where a metadata PG is `unknown`, should we treat declaring OSDs lost as much more dangerous? Could that make the whole filesystem unusable rather than only losing some files? 5. Is there any recommended way to map the affected CephFS data PG to file paths while the PG is still down? `rados ls --pgid` hangs, so we currently cannot list objects in the PG. 6. Before declaring the unavailable OSDs lost, are there any additional diagnostic or recovery steps you would recommend to better assess the situation or reduce the risk of data loss? Any advice would be greatly appreciated. Best regards, Nadir and Alessandro

1 0

pyo3 problem
by BASSAGET Cédric 22 Jun '26

22 Jun '26

Hello I first saw this problem about 2 years ago on ceph cluster (upgraded from 14 ... to 18) https://tracker.ceph.com/issues/64213 I'm unable to load cephadm module because of "ImportError: PyO3 modules may only be initialized once per interpreter process" error I tried to remove restful and nfs modules but they are still in the mgr config : # ceph daemon mgr.ceph-pa2-2 config show | grep -i modules "mgr_disabled_modules": "", "mgr_initial_modules": "restful iostat nfs", "mgr_standby_modules": "false", and I can't modify this key. I think it's in the ceph global config set by a previous ceph version (nothing in ceph.conf) # ceph-conf -D | grep module mgr_disabled_modules = mgr_initial_modules = restful iostat nfs mgr_module_path = /usr/share/ceph/mgr mgr_standby_modules = true I managed to fix the problem on another ceph 18 cluster many days of struggle but can't fix it on this cluster. here are the loaded modules : # ceph mgr module ls | grep on balancer on (always on) crash on (always on) devicehealth on (always on) orchestrator on (always on) pg_autoscaler on (always on) progress on (always on) rbd_support on (always on) status on (always on) telemetry on (always on) volumes on (always on) cephadm on prometheus on I'm running ceph 18.2.8 on debian 12. the aim of making cephadm work is to switch from legacy to orch / cephadm and upgrade to 20. should I consider upgrading my legacy cluster to tentacle and after that switching to cephadm ? Thanks for your help... Regards Cédric

1 2

Re: mgmt-gateway - Issue with Prometheus/Alertmanager mTLS
by Gilles Mocellin 20 Jun '26

20 Jun '26

Le vendredi 19 juin 2026, 12:50:52 heure d’été d’Europe centrale Cameron Ditchfield via ceph-users a écrit : > Hi ! > > Has anyone running Tentacle successfully got the management gateway working > with a non-cephadm-root issued certificate? > > I'm struggling to get it working with Prometheus & Alertmanager when using > a cert not issued by the cephadm-root CA. I have a Tentacle cluster v20.2.1 > that has the management gateway deployed across three nodes with a > keepalived ingress and virtual IP. I've updated the dashboard to point to > the DNS hostname for the VIP for alertmanager and prometheus. > > I can access the dashboard via the hostname, as expected, but there are no > metrics on the overview page and there's a cannot verify SSL cert error for > both alertmanager and prometheus. The CA that issued the gateway cert is > trusted by all nodes in the cluster. > > Thanks, > Cameron Yes, but I think I had and still have some problems : See here https://tracker.ceph.com/issues/74164 I've still not upgraded my LAB to 20.2.2, but it doesn't seems that the tracker issue has progressed...

1 0

rbd config pool set: librbd::PoolMetadata: update_pool_timestamp: (13) Permission denied
by Konstantin Shalygin 19 Jun '26

19 Jun '26

Hi, When rbd client try's to set RBD per pool QoS settings, the settings set's successfully, but also rbd prints this message [root@api1]# rbd --id rbd_user config pool set rbd.example-1 rbd_qos_read_bps_limit 20001 2026-06-19T13:18:30.623+0000 7ff12d46a3c0 -1 librbd::PoolMetadata: update_pool_timestamp: failed to notify clients of pool config update: (13) Permission denied Seems this is part of code [1] The caps is: client.rbd_user key: <snip></snap> caps: [mgr] allow command "service status" caps: [mon] profile rbd, allow command "osd blacklist", allow command "osd blocklist" caps: [osd] profile rbd The question is: * should non-admin account allowed to write update_pool_timestamp [by design]? * should 'profile rbd' be allowed to write update_pool_timestamp [dev's just forget add this to profile]? * can be possible 'allow command' for this via current caps? Thanks, k [1] https://github.com/ceph/ceph/blob/main/src/librbd/api/PoolMetadata.cc#L23-L…

1 0

mgmt-gateway - Issue with Prometheus/Alertmanager mTLS
by Cameron Ditchfield 19 Jun '26

19 Jun '26

Hi ! Has anyone running Tentacle successfully got the management gateway working with a non-cephadm-root issued certificate? I'm struggling to get it working with Prometheus & Alertmanager when using a cert not issued by the cephadm-root CA. I have a Tentacle cluster v20.2.1 that has the management gateway deployed across three nodes with a keepalived ingress and virtual IP. I've updated the dashboard to point to the DNS hostname for the VIP for alertmanager and prometheus. I can access the dashboard via the hostname, as expected, but there are no metrics on the overview page and there's a cannot verify SSL cert error for both alertmanager and prometheus. The CA that issued the gateway cert is trusted by all nodes in the cluster. Thanks, Cameron -- Thought Machine Group Limited, a company registered in England & Wales. Registered number: 11114277. Registered Office: 5 New Street Square, London EC4A 3TW <https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW&entry=gmail…>. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.

1 0

Query regarding Debian Trixie support for Squid and Tentacle
by Piradeep G 19 Jun '26

19 Jun '26

Hi everyone, I have a question regarding package availability for Debian Trixie. Currently, I can see that the Debian-Squid packages are available only for the Bookworm and Jammy distributions. Are there any plans to provide Debian-Squid packages for Debian Trixie as well? Similarly, I noticed that the Debian-Tentacle packages are currently available only for Bookworm, Jammy, and Noble. Is support for Debian Trixie planned for Debian-Tentacle, and if so, is there an estimated timeline for when it will be available? We're evaluating our upgrade plans and would appreciate any information or insights from the community. Thanks in advance! Regards, Piradeep G

1 0

Re: MDS: uh oh, unknown log event type
by Venky Shankar 19 Jun '26

19 Jun '26

Hi Eugen, On Fri, Jun 19, 2026 at 2:27 PM Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I am currently looking into MDS warnings (behind on trimming) on a > customer cluster. They recently upgraded to Squid 19.2.3 which seems > to have fixed a bug with trimming, so the warning doesn't come that > often anymore (if at all currently). But when digging a bit deeper, I > noticed these messages in the event get list output: > > cephfs-journal-tool --rank=cephfs:0 event get list | head > 2026-06-18T13:12:22.821+0000 7fec1b5ff280 0 uh oh, unknown log event > type 100 length 14 > 2026-06-18T13:12:22.833+0000 7fec1b5ff280 0 uh oh, unknown log event > type 100 length 14 What version is cephfs-journal-tool (--version) and is it the same version as (or close to) the ceph cluster? > > That message is repeated 118 times or so before dumping the actual > event list. I looked at the code for 19.2.0 [0] and 19.2.3 [1], in > both cases event type 100 seems to be defined: > > #define EVENT_SEGMENT 100 > > So why is it unknown? It doesn't appear to have any negative impact, > at least noone has complained yet, but I still wanted to ask what this > is about. Can I ignore this? Will that go away on its own at some point? > > Any pointers are appreciated! > > Thanks! > Eugen > > [0] https://github.com/ceph/ceph/blob/v19.2.0/src/mds/LogEvent.h#L45 > [1] https://github.com/ceph/ceph/blob/v19.2.3/src/mds/LogEvent.h#L45 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

MDS: uh oh, unknown log event type
by Eugen Block 19 Jun '26

19 Jun '26

Hi, I am currently looking into MDS warnings (behind on trimming) on a customer cluster. They recently upgraded to Squid 19.2.3 which seems to have fixed a bug with trimming, so the warning doesn't come that often anymore (if at all currently). But when digging a bit deeper, I noticed these messages in the event get list output: cephfs-journal-tool --rank=cephfs:0 event get list | head 2026-06-18T13:12:22.821+0000 7fec1b5ff280 0 uh oh, unknown log event type 100 length 14 2026-06-18T13:12:22.833+0000 7fec1b5ff280 0 uh oh, unknown log event type 100 length 14 That message is repeated 118 times or so before dumping the actual event list. I looked at the code for 19.2.0 [0] and 19.2.3 [1], in both cases event type 100 seems to be defined: #define EVENT_SEGMENT 100 So why is it unknown? It doesn't appear to have any negative impact, at least noone has complained yet, but I still wanted to ask what this is about. Can I ignore this? Will that go away on its own at some point? Any pointers are appreciated! Thanks! Eugen [0] https://github.com/ceph/ceph/blob/v19.2.0/src/mds/LogEvent.h#L45 [1] https://github.com/ceph/ceph/blob/v19.2.3/src/mds/LogEvent.h#L45

1 0

Re: lvm / udev regression, was Re: Re: v20.2.2 Tentacle released
by Guillaume ABRIOUX 18 Jun '26

18 Jun '26

Hi, PR #67989 intentionally stopped excluding all LVM mapper devices so that public LVs show up in 'ceph-volume inventory --list-all'. The regression was that we dropped both the RuntimeError guard and any filtering of LVM internal RAID legs (_rmeta_/_rimage_/ etc.), which have dm nodes in /sys/block but no standalone udev node The follow up PF (tracker #77486) addresses this without reverting #67989: 1/ in get_devices(), it catches RuntimeError from UdevData and skips the device. 2/ it sskips LVM internal RAID/thin-raid legs via is_internal_lv (_rmeta_/_rimage_/_rtmeta_/_rtimage_ and DM_LV_LAYER). 3/ + skip LVs whose preferred path is not an accessible block device. 4/ finally, in Device._parse(), it doesn't rewrite the path to a missing /dev/vg/lv. Public LVs with a resolvable node remain included, only internal/hidden legs and broken udev paths are dropped. The tests cover internal RAID exclusion and missing node exclusion. Regards, -- Guillaume Abrioux Software Engineer From: Harry G Coin via ceph-users <ceph-users(a)ceph.io> Date: Wednesday, 17 June 2026 at 18:19 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] lvm / udev regression, was Re: Re: v20.2.2 Tentacle released Confirming Reto's report and adding the root cause, verified against the diff. That regression hit me as well. Basically, using the lvm system for anything else on the same host that uses ceph wedges. For example, booting the host that uses ceph... Detail: The regression is PR #67989 ("ceph-volume: include LVM mapper devices in get_devices()"). It removed this block from get_devices() in ceph_volume/util/disk.py: python # If the mapper device is a logical volume it gets excluded try: if UdevData(diskname).is_lvm: continue except RuntimeError: logger.debug("get_devices(): device {} couldn't be found.".format(diskname)) continue That block did two things, both now gone: it skipped LVM devices, and — decisively — it caught the RuntimeError that UdevData.__init__ raises when a device path has no udev node. RAID-internal sub-LVs (<lv>_rmeta_N, <lv>_rimage_N) have no standalone udev node. Post-PR, get_devices() calls UdevData() on every dm node including those sub-LVs, the RuntimeError is uncaught, and the whole inventory aborts — hence CEPHADM_REFRESH_FAILED on any host with an LVM RAID1 (or RAID5/6/10) LV. The PR's added test (test_lvm_device_is_included) only covers an LV with a resolvable udev node; the raising path is untested. Trigger condition: present on v20.2.2; absent on v20.2.1 (where the exclude-and-catch block is still in place — verified here, inventory clean on a host with multiple RAID1 LVs). Any host whose OS/boot mirror uses LVM RAID1 is affected; the run aborts on the first RAID-internal sub-LV enumerated. LVM filter/global_filter is not a workaround — those act on PV device paths, not LV names, so a reject of *_rmeta_*/*_rimage_* leaves the sub-LVs fully visible to ceph-volume's /sys/block walk. Suggested fix: restore handling for LVs whose UdevData can't resolve a udev node — re-add the except RuntimeError: continue, or skip LVs whose LVM role is RAID-internal (the [bracketed], private,raid,metadata/image LVs) during enumeration. Reto's "only enumerate public-role LVs" framing is the same idea. Workaround until fixed: stay on v20.2.1 for hosts with RAID-backed OS mirrors. On 6/17/26 05:07, Reto Gysi via ceph-users wrote: > Hi > > I've noticed the following bug/regression with Tentacle release v20.2.2 > Ceph health checks are failing because there are existing raid mirror > volumes on the node hosts, and ceph-volume is trying to read the "raid > internal" logical volumes > > [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices > host zephir `cephadm ceph-volume` failed: cephadm exited with an error > code: 1, stderr: Inferring config > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config > Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep > h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e > CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v > /var/log/ceph/27923302-87a5-11e > c-ac5b-976d21a49941:/var/log/ceph:z -v > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v > /run/udev:/run/ude > v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v > /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 > inventory > --format=json-pretty --with-lsm --filter-for-batch > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr Traceback (most recent call last): > /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> > /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', > 'console_scripts', 'ceph-volume')()) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__ > /usr/bin/podman: stderr self.main(self.argv) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in > newfunc > /usr/bin/podman: stderr return f(*a, **kw) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main > /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in > dispatch > /usr/bin/podman: stderr instance.main() > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, > in main > /usr/bin/podman: stderr > self.format_report(Devices(filter_for_batch=self.args.filter_for_batch, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in > __init__ > /usr/bin/podman: stderr self._devices = [Device(k, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in > <listcomp> > /usr/bin/podman: stderr self._devices = [Device(k, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in > __init__ > /usr/bin/podman: stderr self.device_id = self._get_device_id() > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in > _get_device_id > /usr/bin/podman: stderr udev_data = disk.UdevData(self.path) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in > __init__ > /usr/bin/podman: stderr raise RuntimeError(f'{path} not found.') > /usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found. > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 5296, in <module> > main() > ~~~~^^ > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 5284, in main > r = ctx.func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 397, in _infer_config > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 312, in _infer_fsid > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 425, in _infer_image > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 299, in _validate_fsid > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 3231, in command_ceph_volume > out, err, code = call_throws(ctx, c.run_cmd(), > verbosity=CallVerbosity.QUIET_UNLESS_ERROR) > > ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", > line 310, in call_throws > raise RuntimeError('Failed command: %s' % ' '.join(command)) > RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io > /ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes > -e CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v > /var/log/ceph/27923302-87a5 > -11ec-ac5b-976d21a49941:/var/log/ceph:z -v > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run > /udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v > /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven > tory --format=json-pretty --with-lsm --filter-for-batch > > > lvs output example: > root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o > +devices| grep var > LV VG > Attr LSize Pool Origin > Data% Meta% Move Log Cpy%Sync Convert Role > Devices > > var debian > rwi-aor--- <139.70g > 100.00 > public > var_rimage_0(0),var_rimage_1(0) > > var_backup debian > -wi-ao---- 120.00g > > public > > /dev/mapper/sdc1_crypt(23040) > > [var_rimage_0] debian > iwi-aor--- <139.70g > > private,ra > id,image /dev/mapper/nvme0n1p3_crypt(71525) > > [var_rimage_1] debian > iwi-aor--- <139.70g > > private,ra > id,image /dev/mapper/nvme1n1p1_crypt(214578) > > [var_rmeta_0] debian > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme0n1p3_crypt(0) > > [var_rmeta_1] debian > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme1n1p1_crypt(214577) > > var-lib-ceph debian-4k > rwi-aor--- 40.00g > 100.00 > public > var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0) > > [var-lib-ceph_rimage_0] debian-4k > iwi-aor--- 40.00g > > private,ra > id,image /dev/mapper/nvme3n1_crypt(55050) > > [var-lib-ceph_rimage_1] debian-4k > iwi-aor--- 40.00g > > private,ra > id,image /dev/mapper/nvme4n1_crypt(786433) > > [var-lib-ceph_rmeta_0] debian-4k > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme3n1_crypt(55049) > > [var-lib-ceph_rmeta_1] debian-4k > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme4n1_crypt(786432) > > root@zephir:~# > > Maybe ceph-volume should only try to read the logical volumes with Role > 'public' and not the lv raid internal ones. > Those volumes are used to host the standard OS installation directories > (/,/var,etc) on mirrored volumes. > > >> ceph-volume >> ----------- >> >> * Inventory Scanning: Enhanced inventory discovery logic to >> automatically skip RAM disk devices (/dev/ram*). >> >> To unsubscribe send an email toceph-users-leave(a)ceph.io > > > Thanks & Regards > > Reto > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: 20.2.2 breaks management gateway
by Redouane Kachach 18 Jun '26

18 Jun '26

Thanks for reporting the regression, the PR https://github.com/ceph/ceph/pull/69574 is meant to fix the issue. Best Regards, Redouane. On Wed, Jun 17, 2026 at 9:09 PM Salatiel Filho via ceph-users < ceph-users(a)ceph.io> wrote: > There is a regression which changed the dashboard bind address from > *:8443 to $mon_ip:8443 > That broke the management gateway if its VIP is not in the same mon_ip > range. > All upstreams will become unavailable now. > > > Atenciosamente/Kind regards, > Salatiel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: 20.2.2 breaks management gateway
by Afreen 18 Jun '26

18 Jun '26

Raised a fix https://github.com/ceph/ceph/pull/69574 On Thu, Jun 18, 2026 at 12:41 AM Salatiel Filho via ceph-users < ceph-users(a)ceph.io> wrote: > There is a regression which changed the dashboard bind address from > *:8443 to $mon_ip:8443 > That broke the management gateway if its VIP is not in the same mon_ip > range. > All upstreams will become unavailable now. > > > Atenciosamente/Kind regards, > Salatiel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

20.2.2 breaks management gateway
by Salatiel Filho 18 Jun '26

18 Jun '26

There is a regression which changed the dashboard bind address from *:8443 to $mon_ip:8443 That broke the management gateway if its VIP is not in the same mon_ip range. All upstreams will become unavailable now. Atenciosamente/Kind regards, Salatiel

1 0

Experiences migrating conventional CRUSH rules to MSR at scale?
by Jane Zhu (BLOOMBERG/ 120 PARK) 18 Jun '26

18 Jun '26

Hi all, We're planning a migration from conventional CRUSH rules (chooseleaf firstn/indep) to MSR rules (choosemsr) on large scale production clusters (thousands of OSDs). We've validated the procedure on a test cluster using pg_upmap pinning to avoid data movement during the rule change, then releasing pins in batches to control backfill. I'd appreciate hearing from anyone who has done this at scale. A few specific areas we're trying to get more data on: 1. Peering storm on rule change — changing a pool's crush_rule triggers re-peering for pretty much every PG in that pool. How long did peering take to settle on your clusters, and was client I/O meaningfully impacted? 2. OSDMap bloat from massive pg_upmap — did you hit any issues with OSD map distribution lag, OSDs falling behind on epochs, or MON store growth? Did you batch the upmap application, and if so, what batch size worked well? 3. pg_upmap apply rate — how fast can ceph osd pg-upmap commands be applied in practice at scale? Did you see any MON bottlenecks when applying tens of thousands of upmaps in rapid succession? 4. Pin release strategy — When releasing pg_upmap pins to allow data movement to the new MSR placement, what batch size and pacing worked for you? 5. Anything else we should watch for? For context, in our testing on a Squid cluster, we confirmed that pg_upmap_items does NOT survive a rule change (they are pruned from the OSDMap), so we're using pg_upmap (full acting set override). Thanks in advance for any data points or stories! -Jane

1 0

lvm / udev regression, was Re: Re: v20.2.2 Tentacle released
by Harry G Coin 18 Jun '26

18 Jun '26

Confirming Reto's report and adding the root cause, verified against the diff. That regression hit me as well. Basically, using the lvm system for anything else on the same host that uses ceph wedges. For example, booting the host that uses ceph... Detail: The regression is PR #67989 ("ceph-volume: include LVM mapper devices in get_devices()"). It removed this block from get_devices() in ceph_volume/util/disk.py: python # If the mapper device is a logical volume it gets excluded try: if UdevData(diskname).is_lvm: continue except RuntimeError: logger.debug("get_devices(): device {} couldn't be found.".format(diskname)) continue That block did two things, both now gone: it skipped LVM devices, and — decisively — it caught the RuntimeError that UdevData.__init__ raises when a device path has no udev node. RAID-internal sub-LVs (<lv>_rmeta_N, <lv>_rimage_N) have no standalone udev node. Post-PR, get_devices() calls UdevData() on every dm node including those sub-LVs, the RuntimeError is uncaught, and the whole inventory aborts — hence CEPHADM_REFRESH_FAILED on any host with an LVM RAID1 (or RAID5/6/10) LV. The PR's added test (test_lvm_device_is_included) only covers an LV with a resolvable udev node; the raising path is untested. Trigger condition: present on v20.2.2; absent on v20.2.1 (where the exclude-and-catch block is still in place — verified here, inventory clean on a host with multiple RAID1 LVs). Any host whose OS/boot mirror uses LVM RAID1 is affected; the run aborts on the first RAID-internal sub-LV enumerated. LVM filter/global_filter is not a workaround — those act on PV device paths, not LV names, so a reject of *_rmeta_*/*_rimage_* leaves the sub-LVs fully visible to ceph-volume's /sys/block walk. Suggested fix: restore handling for LVs whose UdevData can't resolve a udev node — re-add the except RuntimeError: continue, or skip LVs whose LVM role is RAID-internal (the [bracketed], private,raid,metadata/image LVs) during enumeration. Reto's "only enumerate public-role LVs" framing is the same idea. Workaround until fixed: stay on v20.2.1 for hosts with RAID-backed OS mirrors. On 6/17/26 05:07, Reto Gysi via ceph-users wrote: > Hi > > I've noticed the following bug/regression with Tentacle release v20.2.2 > Ceph health checks are failing because there are existing raid mirror > volumes on the node hosts, and ceph-volume is trying to read the "raid > internal" logical volumes > > [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices > host zephir `cephadm ceph-volume` failed: cephadm exited with an error > code: 1, stderr: Inferring config > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config > Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep > h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e > CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v > /var/log/ceph/27923302-87a5-11e > c-ac5b-976d21a49941:/var/log/ceph:z -v > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v > /run/udev:/run/ude > v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v > /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 > inventory > --format=json-pretty --with-lsm --filter-for-batch > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No > such file or directory > /usr/bin/podman: stderr Traceback (most recent call last): > /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> > /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', > 'console_scripts', 'ceph-volume')()) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__ > /usr/bin/podman: stderr self.main(self.argv) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in > newfunc > /usr/bin/podman: stderr return f(*a, **kw) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main > /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in > dispatch > /usr/bin/podman: stderr instance.main() > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, > in main > /usr/bin/podman: stderr > self.format_report(Devices(filter_for_batch=self.args.filter_for_batch, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in > __init__ > /usr/bin/podman: stderr self._devices = [Device(k, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in > <listcomp> > /usr/bin/podman: stderr self._devices = [Device(k, > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in > __init__ > /usr/bin/podman: stderr self.device_id = self._get_device_id() > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in > _get_device_id > /usr/bin/podman: stderr udev_data = disk.UdevData(self.path) > /usr/bin/podman: stderr File > "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in > __init__ > /usr/bin/podman: stderr raise RuntimeError(f'{path} not found.') > /usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found. > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 5296, in <module> > main() > ~~~~^^ > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 5284, in main > r = ctx.func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 397, in _infer_config > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 312, in _infer_fsid > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 425, in _infer_image > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 299, in _validate_fsid > return func(ctx) > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", > line 3231, in command_ceph_volume > out, err, code = call_throws(ctx, c.run_cmd(), > verbosity=CallVerbosity.QUIET_UNLESS_ERROR) > > ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File > "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", > line 310, in call_throws > raise RuntimeError('Failed command: %s' % ' '.join(command)) > RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io > /ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes > -e CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v > /var/log/ceph/27923302-87a5 > -11ec-ac5b-976d21a49941:/var/log/ceph:z -v > /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run > /udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v > /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven > tory --format=json-pretty --with-lsm --filter-for-batch > > > lvs output example: > root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o > +devices| grep var > LV VG > Attr LSize Pool Origin > Data% Meta% Move Log Cpy%Sync Convert Role > Devices > > var debian > rwi-aor--- <139.70g > 100.00 > public > var_rimage_0(0),var_rimage_1(0) > > var_backup debian > -wi-ao---- 120.00g > > public > > /dev/mapper/sdc1_crypt(23040) > > [var_rimage_0] debian > iwi-aor--- <139.70g > > private,ra > id,image /dev/mapper/nvme0n1p3_crypt(71525) > > [var_rimage_1] debian > iwi-aor--- <139.70g > > private,ra > id,image /dev/mapper/nvme1n1p1_crypt(214578) > > [var_rmeta_0] debian > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme0n1p3_crypt(0) > > [var_rmeta_1] debian > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme1n1p1_crypt(214577) > > var-lib-ceph debian-4k > rwi-aor--- 40.00g > 100.00 > public > var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0) > > [var-lib-ceph_rimage_0] debian-4k > iwi-aor--- 40.00g > > private,ra > id,image /dev/mapper/nvme3n1_crypt(55050) > > [var-lib-ceph_rimage_1] debian-4k > iwi-aor--- 40.00g > > private,ra > id,image /dev/mapper/nvme4n1_crypt(786433) > > [var-lib-ceph_rmeta_0] debian-4k > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme3n1_crypt(55049) > > [var-lib-ceph_rmeta_1] debian-4k > ewi-aor--- 4.00m > > private,ra > id,metadata /dev/mapper/nvme4n1_crypt(786432) > > root@zephir:~# > > Maybe ceph-volume should only try to read the logical volumes with Role > 'public' and not the lv raid internal ones. > Those volumes are used to host the standard OS installation directories > (/,/var,etc) on mirrored volumes. > > >> ceph-volume >> ----------- >> >> * Inventory Scanning: Enhanced inventory discovery logic to >> automatically skip RAM disk devices (/dev/ram*). >> >> To unsubscribe send an email toceph-users-leave(a)ceph.io > > > Thanks & Regards > > Reto > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Re: v20.2.2 Tentacle released
by Reto Gysi 17 Jun '26

17 Jun '26

Hi I've noticed the following bug/regression with Tentacle release v20.2.2 Ceph health checks are failing because there are existing raid mirror volumes on the node hosts, and ceph-volume is trying to read the "raid internal" logical volumes [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host zephir `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/mon.zephir/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/cep h/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v /var/log/ceph/27923302-87a5-11e c-ac5b-976d21a49941:/var/log/ceph:z -v /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/ude v -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inventory --format=json-pretty --with-lsm --filter-for-batch /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr stderr: blkid: error: /dev/debian/var_rmeta_0: No such file or directory /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module> /usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__ /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, in main /usr/bin/podman: stderr self.format_report(Devices(filter_for_batch=self.args.filter_for_batch, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in __init__ /usr/bin/podman: stderr self._devices = [Device(k, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in <listcomp> /usr/bin/podman: stderr self._devices = [Device(k, /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in __init__ /usr/bin/podman: stderr self.device_id = self._get_device_id() /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in _get_device_id /usr/bin/podman: stderr udev_data = disk.UdevData(self.path) /usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in __init__ /usr/bin/podman: stderr raise RuntimeError(f'{path} not found.') /usr/bin/podman: stderr RuntimeError: /dev/debian/var_rmeta_0 not found. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5296, in <module> main() ~~~~^^ File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5284, in main r = ctx.func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 397, in _infer_config return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 312, in _infer_fsid return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 425, in _infer_image return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 299, in _validate_fsid return func(ctx) File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 3231, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR) ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", line 310, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io /ceph/ceph:v20.2.2 -e NODE_NAME=zephir -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/27923302-87a5-11ec-ac5b-976d21a49941:/var/run/ceph:z -v /var/log/ceph/27923302-87a5 -11ec-ac5b-976d21a49941:/var/log/ceph:z -v /var/lib/ceph/27923302-87a5-11ec-ac5b-976d21a49941/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run /udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpe1ewv2jz:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.2.2 inven tory --format=json-pretty --with-lsm --filter-for-batch lvs output example: root@zephir:~# lvs -a -o +role -o +devices |head -1; lvs -a -o +role -o +devices| grep var LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Role Devices var debian rwi-aor--- <139.70g 100.00 public var_rimage_0(0),var_rimage_1(0) var_backup debian -wi-ao---- 120.00g public /dev/mapper/sdc1_crypt(23040) [var_rimage_0] debian iwi-aor--- <139.70g private,ra id,image /dev/mapper/nvme0n1p3_crypt(71525) [var_rimage_1] debian iwi-aor--- <139.70g private,ra id,image /dev/mapper/nvme1n1p1_crypt(214578) [var_rmeta_0] debian ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme0n1p3_crypt(0) [var_rmeta_1] debian ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme1n1p1_crypt(214577) var-lib-ceph debian-4k rwi-aor--- 40.00g 100.00 public var-lib-ceph_rimage_0(0),var-lib-ceph_rimage_1(0) [var-lib-ceph_rimage_0] debian-4k iwi-aor--- 40.00g private,ra id,image /dev/mapper/nvme3n1_crypt(55050) [var-lib-ceph_rimage_1] debian-4k iwi-aor--- 40.00g private,ra id,image /dev/mapper/nvme4n1_crypt(786433) [var-lib-ceph_rmeta_0] debian-4k ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme3n1_crypt(55049) [var-lib-ceph_rmeta_1] debian-4k ewi-aor--- 4.00m private,ra id,metadata /dev/mapper/nvme4n1_crypt(786432) root@zephir:~# Maybe ceph-volume should only try to read the logical volumes with Role 'public' and not the lv raid internal ones. Those volumes are used to host the standard OS installation directories (/,/var,etc) on mirrored volumes. > > ceph-volume > ----------- > > * Inventory Scanning: Enhanced inventory discovery logic to > automatically skip RAM disk devices (/dev/ram*). > > To unsubscribe send an email to ceph-users-leave(a)ceph.io Thanks & Regards Reto

1 0

Re: v20.2.2 Tentacle released (incomplete EU mirror sync)
by Florian Haas 17 Jun '26

17 Jun '26

Hello, it looks like after the 20.2.2 release, the sync to the EU package mirror was incomplete. Compare: https://eu.ceph.com/debian-tentacle/pool/main/c/ceph/ to https://download.ceph.com/debian-tentacle/pool/main/c/ceph/ Apparently the contents of https://eu.ceph.com/debian-tentacle/dists/noble/ *were* fully replicated though, so this means that the EU mirror now contains an index that refers to .deb files that don't exist there, leading to 404s on "apt upgrade" if this is the only Ceph mirror configured. This issue can be worked around by reconfiguring one's APT sources to download.ceph.com, but it would be great if the EU mirror could also be fixed. Thanks! Cheers, Florian On 16/06/2026 21:51, Patrick Donnelly via ceph-users wrote: > Greetings, > > This is the second minor release in the Tentacle series. > > We recommend that all users update to this release.

1 0

Re: v20.2.2 Tentacle released
by Robert Sander 17 Jun '26

17 Jun '26

Hi, Am 16.06.26 um 9:52 PM schrieb Patrick Donnelly via ceph-users: > ceph-volume > ----------- > > * Inventory Scanning: Enhanced inventory discovery logic to > automatically skip RAM disk devices (/dev/ram*). Why was https://tracker.ceph.com/issues/76043 not included in this release? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

v20.2.2 Tentacle released
by Patrick Donnelly 17 Jun '26

17 Jun '26

Greetings, This is the second minor release in the Tentacle series. We recommend that all users update to this release. Notable Changes --------------- Rocky 10 package-based installs are now supported starting with v20.2.2. Please see the supported platforms ( https://docs.ceph.com/en/latest/start/os-recommendations/#platforms ) or current and planned support in Ceph. MDS (Metadata Server) --------------------- * Segmentation fault fixed due to incorrect queueing of request retries. OSD (Object Storage Daemon) --------------------------- * PGLog Missed List: Fixed a bug to ensure the correct version is attached to the missing list when ignoring log entries. * Data Integrity Asserts: Added assertions to explicitly catch potential corruption in the OSD missing list. RGW (RADOS Gateway) ------------------- * Lifecycle Management: Fixed lifecycle transition issues affecting encrypted multipart objects. * REST & Query Handling: RESTArgs::get_string() now properly URL-decodes incoming query parameters. RADOS / librados / neorados --------------------------- * Linger Operations: Rewrote safety checks to eliminate use-after-free vulnerabilities and LingerOp memory leaks when an unwatch operation returns ENOTCONN. Replaced the flawed is_valid_watch() check with a safe linger_by_cookie() lookup which safely manages LingerOp references using intrusive_ptr. Ensured librados linger callbacks hold a persistent reference to LingerOp to protect against races with simultaneous linger_cancel() requests. Configured librados::IoCtxImpl::aio_unwatch() to asynchronously deliver ENOTCONN to AioCompletion instead of returning the error directly. * Watch/Notify: Fixed neorados notification queue bounds enforcement so that an overflow marker is only appended on the first message exceeding capacity, rather than duplicating it on every subsequent message. Prevented double-cleanup triggers in watch/notify operations when incoming errors arrive after maybe_cleanup() runs. Fixed a bug where notify would lose original error values by ensuring it no longer attempts to decode empty responses. Enhanced io_context shutdown procedures to clear handlers and route cleanly through linger_cancel to avoid use-after-free states. * Async Utilities: Fixed an issue regarding the improper removal of objects from the async service list. Dashboard ----------------- * NVMeoF Revamp: Complete UI overhaul with DHCHAP controller key support, namespace encryption, and secure listeners configuration. * Pools & RGW: Added stretch cluster validation for pools, fixed RGW restart/stop issues, storage class restrictions, sync policy fixes, and MSR EC Profile support. * NFS: Toggle visibility for CephFS snapshots, fixed export creation and path value consistency issues ceph-volume ----------- * Inventory Scanning: Enhanced inventory discovery logic to automatically skip RAM disk devices (/dev/ram*). External Block Device (extblkdev) --------------------------------- * Plugin Stability: Fixed an assertion failure in the FCM plugin when encountering multivolume devices. Testing & QA (Suites / Tooling) ------------------------------- * OS Testing: Added CentOS 9 Stream as a supported host OS for the tentacle-p2p stress-split suite. * Dependency & Compilation: Configured bare-metal upgrade testing workloads (rbd-python) to inject python3-pytest via system packages to prevent test failures. Patched the blogbench workunit in qa/cephfs to gracefully handle "implicit declaration of function" warnings generated by newer GCC versions. Updated Lua scripts in qa/cephfs to respect missing kernels in the YAML configurations. * Boto3 Migration: Replaced outdated boto2 dependencies with boto3 across qa/radosgw_admin and qa/radosgw_admin_rest test frameworks, removing stale request log components and ACL test cases. * Tools: Resolved a crash occurring in tools/ceph-kvstore-tool during database closures, reduced BlueStore header exposure, and added missing compiler guards (WITH_BLUESTORE). We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-2-tentacle-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.2.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 0fcffee29411e3a38036764817b6e1afc59741cc -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: tentacle 20.2.2 QE validation status
by Venky Shankar 16 Jun '26

16 Jun '26

Hi Yuri, On Wed, Jun 10, 2026 at 10:56 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > Still seeking approvals: > > fs: Venky fs approved. Failures are https://tracker.ceph.com/projects/cephfs/wiki/Tentacle#Tentacle-v2022 > orch/ceph-volume: Redo & Guillaume > nvmeof: Aviv, Vallari > > > On Tue, Jun 9, 2026 at 1:14 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: >> >> Rados approved: https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-additional-PR-a… >> ________________________________ >> From: Yuri Weinstein via dev <dev(a)ceph.io> >> Sent: Monday, June 8, 2026 11:54 PM >> To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> >> Subject: [EXTERNAL] Re: tentacle 20.2.2 QE validation status >> >> We merged several PRs listed here https: //tracker. ceph. com/issues/77140 and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: Aviv, Vallari upgrade: >> We merged several PRs listed here https://tracker.ceph.com/issues/77140 and re-ran several suites. >> Seeking approvals: >> >> rados: Radek, Nitzan >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Redo & Guillaume >> rbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick, Nitzan >> >> On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75878#note-6 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA -- Cheers, Venky

1 0

Re: How to build ceph containers?
by John Mulligan 13 Jun '26

13 Jun '26

On Friday, June 12, 2026 7:49:18 AM Eastern Daylight Time Adam Prycki via ceph-users wrote: > I have a newbie question. > > How does one build ceph container which can be tested in cephadm cluster? > > I've had success compiling ceph with this command from the documentation > ./src/script/build-with-container.py -d ubuntu22.04 -b build.u2204 -e build > > but container it leaves behind has no ceph binaries in $PATH or /usr/bin. > That's intentional. The container that script produces is meant purely to build ceph. To produce a container for running ceph - one first needs to build the (rpm) packages. You can use the build-with-container.py script and the -e packages with a distro base of rocky10 (`-d rocky10 -e packages`) or centos 9 (`-d centos9 -e packages`), depending on what ceph branch you are targeting. Once you have packages you need to host them in a yum/dnf repo. Then you can use the script ./container/build.sh with the special env var CUSTOM_CEPH_REPO_URL set to your new repo. It's been a few months since I last tried it myself, but that's the process I used. We've batted around the idea of teaching build-with-container.py how to help automate some of the steps after package build (like creating the repo and running a local webserver to host it) - but nothing concrete has really come of that so far. > I was testing on ubunut 22.04 with podman and v19.2.4 git tag. > As of today, we regularly build packages for centos 9, rocky 10, Ubuntu 22.04 (jammy), and Ubuntu 24.04 (noble) on main. The container/Containerfile builds the images based on EL distro base - centos9 and rocky 10. The default was recently updated to rocky 10. To use something other than those two - you'd probably be better off creating your own {Contatainerfile,Dockerfile}. > Second question, how distro specific are these container build scripts? > My colleague couldn't build any images on ubuntu 26.04. > I can build only ubnutu 22.04 images, centos9 and centos10 builds fail > on v19.2.4 and v20.2.1 > Ubuntu 26.04 (resolute) support exists in build-with-container.py but it is not being tested in the CI. It's being discussed as a target for a future ceph release so I expect some more activity on that front soon. I don't expect either the basic build or the package build to work at the moment but I have not tried it myself recently. Each branch will have it's own "supported build matrix" so I am unsure how to help with the 2nd sentence there. Maybe more specific, "I used branch X on host os Y and saw Z failure" would help. > Third question, is there a reliable way to set number of jobs used > during compilation? BUILD_MAKEOPTS=-j32 seems to work for some build > targets but not for all. By default build system chooses -j16 on 32 core VM. > Some of the build targets have different behaviors because they're driven by different scripts internally. The stuff that drives the basic build and 'make check' is a bit different from the package builds for instance. You should be able to pass any env var you want via an environment file. Take a look at this example from the ceph CI : https://github.com/ceph/ceph-build/blob/ b8a350545b563843fc96a7e0f0fb2c1fbeb98246/ceph-pull-requests/build/build#L13- L20 But what vars will be honored is dependent on what you run. Also, feel free to file tracker.ceph.com issues for specific problems you may encounter while trying these tools. You can also reach us on Ceph community slack in the #ceph-build-sig channel if you prefer chat over email. We also hold weekly meetings on 4:30PM Eastern on Wednesdays. Check the ceph community calendar for details. Although, I realize that's pretty late for folks in Europe.

1 0

How to build ceph containers?
by Adam Prycki 12 Jun '26

12 Jun '26

I have a newbie question. How does one build ceph container which can be tested in cephadm cluster? I've had success compiling ceph with this command from the documentation ./src/script/build-with-container.py -d ubuntu22.04 -b build.u2204 -e build but container it leaves behind has no ceph binaries in $PATH or /usr/bin. I was testing on ubunut 22.04 with podman and v19.2.4 git tag. Second question, how distro specific are these container build scripts? My colleague couldn't build any images on ubuntu 26.04. I can build only ubnutu 22.04 images, centos9 and centos10 builds fail on v19.2.4 and v20.2.1 Third question, is there a reliable way to set number of jobs used during compilation? BUILD_MAKEOPTS=-j32 seems to work for some build targets but not for all. By default build system chooses -j16 on 32 core VM. Best regards Adam Prycki

1 0

Re: Scrubbing behind after Quincy to Reef upgrade
by Eugen Block 12 Jun '26

12 Jun '26

I feel like this has been discussed multiple times on this list, I just don't have any links at hand. I suspect mclock settings, most likely some things changed between Quincy and Reef. You could fall back to wpq instead of mclock, it's still a general recommendation at the moment, and then see if anyting improves. Zitat von Reed Dier via ceph-users <ceph-users(a)ceph.io>: > Hello all, > > TL;DR is that the number of concurrent PGs scrubbing, deep or > otherwise, has appeared to increase by about 5-10x, while the number > of PGs complaining that they haven't been scrubbed, deep or > otherwise, has continued to tick higher. >> HEALTH_WARN 217 pgs not deep-scrubbed in time; 187 pgs not scrubbed in time > > > Hoping there may be something that I must have missed in release > notes and mailing list that explains why my scrubs both exploded in > concurrency, as well as fell behind after upgrading from quincy > (17.2.9) to reef (18.2.8). > > Non-cephadm, U22.04, rather heterogenous OSD hardware. > Mix of 8T and 2T HDD, as well as 2T SSD. > HDD's have NVMe WAL/DB of various sizes depending on when they were deployed. > Mix of replicated and EC pools, as well as some replicated pools > across different device classes. > > The vast majority of the PG's that are behind on scrubbing are on EC > pools, and the vast majority of that, is our EC82 cephfs pool (40) > that holds the bulk of our stored data, and the other largest pool > is an older EC73 cephfs pool (37). > My quick and dirty approximation based on PGs last scrubbed last month. > >> ceph pg dump | grep 2026-05 | awk '{print $1" "$27}' | grep -v >> periodic | cut -d '.' -f1 | sort | uniq -c >> dumped all >> 1 17 >> 1 20 >> 116 37 >> 224 40 > > > I didn't make any changes to scrub intervals or mclock profiles > before/during/after the upgrade. >> ceph config dump | grep mclock_profile | awk '{print $4}' | uniq -c ; >> 313 balanced >> ceph config dump | grep scrub_interval >> global class:ssd advanced osd_deep_scrub_interval >> 604800.000000 >> mon advanced osd_deep_scrub_interval >> 604800.000000 >> mon.* advanced osd_deep_scrub_interval >> 604800.000000 >> mgr.* advanced osd_deep_scrub_interval >> 604800.000000 >> osd class:hdd advanced osd_deep_scrub_interval >> 604800.000000 >> osd class:ssd advanced osd_deep_scrub_interval >> 604800.000000 >> osd advanced osd_deep_scrub_interval >> 604800.000000 >> osd.* advanced osd_deep_scrub_interval >> 604800.000000 > > I've tried ceph tell osd.$osd osd_max_scrubs $more, which seems to > somewhat momentarily drive the count of > active+clean+scrubbing[+deep] PGs, but doesn't seem to make a > demonstrative difference in terms of getting ahead in the number of > PGs behind (number continues to grow). > I also looked at load15 across OSD hosts, and they don't appear to > be anywhere near the 50% threshold of osd_scrub_load_threshold > either, so I think I can rule that one out for now. > > I'm mostly curious why the change in behavior of concurrent scrubs > ballooning, and yet the number of PGs behind on scrubbing ballooning > as well, without anything actually changing. > And I'm also curious what tunables I can turn to get things back > under control for scrubbing both short and long term as I look > towards getting to squid and 24.04. > Is there an internal mechanism that triggers a deeper scrub during > first deep scrub after upgrading a major release, reef or otherwise? > > Included some graphs of scrub load over the last 60 and 365 day > period to show prior scrub load that only exceedingly rarely ever > generated a PG_NOT_[DEEP_]SCRUBBED warning, > as well as raw load average (smallest cpu count is 16, and it > doesn't even autoscale to 8, so nothing should be complaining there.) > https://imgur.com/a/rixNrCe > > Appreciate any pointers anyone can steer me towards. > Reed

1 0

Scrubbing behind after Quincy to Reef upgrade
by Reed Dier 12 Jun '26

12 Jun '26

Hello all, TL;DR is that the number of concurrent PGs scrubbing, deep or otherwise, has appeared to increase by about 5-10x, while the number of PGs complaining that they haven't been scrubbed, deep or otherwise, has continued to tick higher. > HEALTH_WARN 217 pgs not deep-scrubbed in time; 187 pgs not scrubbed in time Hoping there may be something that I must have missed in release notes and mailing list that explains why my scrubs both exploded in concurrency, as well as fell behind after upgrading from quincy (17.2.9) to reef (18.2.8). Non-cephadm, U22.04, rather heterogenous OSD hardware. Mix of 8T and 2T HDD, as well as 2T SSD. HDD's have NVMe WAL/DB of various sizes depending on when they were deployed. Mix of replicated and EC pools, as well as some replicated pools across different device classes. The vast majority of the PG's that are behind on scrubbing are on EC pools, and the vast majority of that, is our EC82 cephfs pool (40) that holds the bulk of our stored data, and the other largest pool is an older EC73 cephfs pool (37). My quick and dirty approximation based on PGs last scrubbed last month. > ceph pg dump | grep 2026-05 | awk '{print $1" "$27}' | grep -v periodic | cut -d '.' -f1 | sort | uniq -c > dumped all > 1 17 > 1 20 > 116 37 > 224 40 I didn't make any changes to scrub intervals or mclock profiles before/during/after the upgrade. > ceph config dump | grep mclock_profile | awk '{print $4}' | uniq -c ; > 313 balanced > ceph config dump | grep scrub_interval > global class:ssd advanced osd_deep_scrub_interval 604800.000000 > mon advanced osd_deep_scrub_interval 604800.000000 > mon.* advanced osd_deep_scrub_interval 604800.000000 > mgr.* advanced osd_deep_scrub_interval 604800.000000 > osd class:hdd advanced osd_deep_scrub_interval 604800.000000 > osd class:ssd advanced osd_deep_scrub_interval 604800.000000 > osd advanced osd_deep_scrub_interval 604800.000000 > osd.* advanced osd_deep_scrub_interval 604800.000000 I've tried ceph tell osd.$osd osd_max_scrubs $more, which seems to somewhat momentarily drive the count of active+clean+scrubbing[+deep] PGs, but doesn't seem to make a demonstrative difference in terms of getting ahead in the number of PGs behind (number continues to grow). I also looked at load15 across OSD hosts, and they don't appear to be anywhere near the 50% threshold of osd_scrub_load_threshold either, so I think I can rule that one out for now. I'm mostly curious why the change in behavior of concurrent scrubs ballooning, and yet the number of PGs behind on scrubbing ballooning as well, without anything actually changing. And I'm also curious what tunables I can turn to get things back under control for scrubbing both short and long term as I look towards getting to squid and 24.04. Is there an internal mechanism that triggers a deeper scrub during first deep scrub after upgrading a major release, reef or otherwise? Included some graphs of scrub load over the last 60 and 365 day period to show prior scrub load that only exceedingly rarely ever generated a PG_NOT_[DEEP_]SCRUBBED warning, as well as raw load average (smallest cpu count is 16, and it doesn't even autoscale to 8, so nothing should be complaining there.) https://imgur.com/a/rixNrCe Appreciate any pointers anyone can steer me towards. Reed

1 0

Re: tentacle 20.2.2 QE validation status
by Guillaume ABRIOUX 11 Jun '26

11 Jun '26

Hi Yuri, ceph-volume approved Regards, -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Wednesday, 10 June 2026 at 19:21 To: Nitzan Mordechai <nmordech(a)ibm.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: tentacle 20.2.2 QE validation status Still seeking approvals: fs: Venky orch/ceph-volume: Redo & Guillaume nvmeof: Aviv, Vallari On Tue, Jun 9, 2026 at 1:14 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: > Rados approved: > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_proje… > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Monday, June 8, 2026 11:54 PM > *To:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> > *Subject:* [EXTERNAL] Re: tentacle 20.2.2 QE validation status > > We merged several PRs listed here https: //tracker. ceph. com/issues/77140 > and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: > Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: > Aviv, Vallari upgrade: > We merged several PRs listed here https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > and re-ran several suites. > Seeking approvals: > > rados: Radek, Nitzan > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redo & Guillaume > rbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick, Nitzan > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: tentacle 20.2.2 QE validation status
by Yuri Weinstein 11 Jun '26

11 Jun '26

Still seeking approvals: fs: Venky orch/ceph-volume: Redo & Guillaume nvmeof: Aviv, Vallari On Tue, Jun 9, 2026 at 1:14 AM Nitzan Mordechai via dev <dev(a)ceph.io> wrote: > Rados approved: > https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-additional-PR-a… > ------------------------------ > *From:* Yuri Weinstein via dev <dev(a)ceph.io> > *Sent:* Monday, June 8, 2026 11:54 PM > *To:* dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> > *Subject:* [EXTERNAL] Re: tentacle 20.2.2 QE validation status > > We merged several PRs listed here https: //tracker. ceph. com/issues/77140 > and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: > Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: > Aviv, Vallari upgrade: > We merged several PRs listed here https://tracker.ceph.com/issues/77140 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > and re-ran several suites. > Seeking approvals: > > rados: Radek, Nitzan > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redo & Guillaume > rbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick, Nitzan > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75878#note-6 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > >

1 0

June Ceph Tech Talk Recording
by Anthony Middleton 10 Jun '26

10 Jun '26

Hey Ceph Friends, If you missed yesterday's Ceph Tech Talk session on Building Ultra-Resilient Ceph Clusters with Xinnor and OSNexus, here are links to the recording and presentation slides. *Slides: https://ceph.io/assets/pdfs/tech-talks/dual_layer_durability_xinnor.pdf <https://ceph.io/assets/pdfs/tech-talks/dual_layer_durability_xinnor.pdf>* *Recording: https://youtu.be/VqKvBuemX9Q <https://youtu.be/VqKvBuemX9Q>* Our next session is in the works for July 8th on the topic of GPU-initiated IO to Ceph. I'll share more details as things progress. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: CephFS large amounts of dark data
by Anthony D'Atri 10 Jun '26

10 Jun '26

> ). >>> >>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. >>> The layout for most files: >>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" >>> >>> But there is 1.4PiB discrepancy between the pool and the filesystem Do you have scrubs enabled? Which if any non-default config options do you have set? Any undersized or degraded or backfilling PGs? Which Ceph release? Do you have a sizable fraction of small files? If you’re running Squid or earlier or don’t have EC optimizations enabled, even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file will consume 352KB, etc. If I understand those layout options correctly. If that’s what’s going on, going to Tentacle with EC optimizations would gain you some efficiency for files newly [re]written. You could also migrate small files to a replicated pool. >>> which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. >>> >> Hello, >> We had a similar issue last year with a group of users that created and removed files at a very high rate. >> >> Have you read https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… ? >> >> We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). >> >> >> Loïc. > > > -- > _____________________________________________________________ > prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > Department of Experimental High Energy Physics - F9 > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > SI-1001 Ljubljana, Slovenia > Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > ------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: CephFS large amounts of dark data
by Gregory Farnum 10 Jun '26

10 Jun '26

Do you have any snapshots? There's also generally some amount of inflation with EC and small files, but I don't think it should reach (7/5.6=)25%, and since you're using 16MB objects I presume most of your files are large. -Greg On Tue, Jun 9, 2026 at 7:02 AM Andrej Filipčič via ceph-users <ceph-users(a)ceph.io> wrote: > > > thanks for this info, but our purge queue is more or less empty > "purge_queue": { > "pq_executing_ops": 2, > "pq_executing_ops_high_water": 13659, > "pq_executing": 1, > "pq_executing_high_water": 500, > "pq_executed_ops": 127095, > "pq_executed": 50471, > "pq_item_in_journal": 2 > }, > > anyway, I raised the parameters to see if it helps. > > Best, > Andrej > > > On 9. 6. 2026 14:57, Loïc Tortay via ceph-users wrote: > > On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote: > >> > >> Hi, > >> > >> I have a problem with large amounts of dark data/orphans on cephfs > >> data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). > >> > >> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M > >> files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used > >> consistent with 8+3 EC. > >> The layout for most files: > >> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > >> object_size=16777216 pool=cephfs_data_echdd" > >> > >> But there is 1.4PiB discrepancy between the pool and the filesystem > >> which I cannot explain and I suspect there are a lot of orphan > >> objects there. I have run mds scrub on / and ~mdsdir as well. There > >> is some mds damage on some old small files (~400 files), which I do > >> not think it's relevant here. > >> > > Hello, > > We had a similar issue last year with a group of users that created > > and removed files at a very high rate. > > > > Have you read > > https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… > > ? > > > > We increased the purge rate parameters (very) aggressively to get back > > to a comfortable situation (i.e. not a pool w/ near full warnings). > > > > > > Loïc. > > > -- > _____________________________________________________________ > prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si > Department of Experimental High Energy Physics - F9 > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > SI-1001 Ljubljana, Slovenia > Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > ------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: CephFS large amounts of dark data
by Andrej Filipčič 09 Jun '26

09 Jun '26

thanks for this info, but our purge queue is more or less empty "purge_queue": { "pq_executing_ops": 2, "pq_executing_ops_high_water": 13659, "pq_executing": 1, "pq_executing_high_water": 500, "pq_executed_ops": 127095, "pq_executed": 50471, "pq_item_in_journal": 2 }, anyway, I raised the parameters to see if it helps. Best, Andrej On 9. 6. 2026 14:57, Loïc Tortay via ceph-users wrote: > On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote: >> >> Hi, >> >> I have a problem with large amounts of dark data/orphans on cephfs >> data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). >> >> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M >> files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used >> consistent with 8+3 EC. >> The layout for most files: >> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 >> object_size=16777216 pool=cephfs_data_echdd" >> >> But there is 1.4PiB discrepancy between the pool and the filesystem >> which I cannot explain and I suspect there are a lot of orphan >> objects there. I have run mds scrub on / and ~mdsdir as well. There >> is some mds damage on some old small files (~400 files), which I do >> not think it's relevant here. >> > Hello, > We had a similar issue last year with a group of users that created > and removed files at a very high rate. > > Have you read > https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… > ? > > We increased the purge rate parameters (very) aggressively to get back > to a comfortable situation (i.e. not a pool w/ near full warnings). > > > Loïc. -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 0

Re: CephFS large amounts of dark data
by Loïc Tortay 09 Jun '26

09 Jun '26

On 09/06/2026 14:10, Andrej Filipcic via ceph-users wrote: > > Hi, > > I have a problem with large amounts of dark data/orphans on cephfs data > pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). > > There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M > files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used > consistent with 8+3 EC. > The layout for most files: > ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > object_size=16777216 pool=cephfs_data_echdd" > > But there is 1.4PiB discrepancy between the pool and the filesystem > which I cannot explain and I suspect there are a lot of orphan objects > there. I have run mds scrub on / and ~mdsdir as well. There is some mds > damage on some old small files (~400 files), which I do not think it's > relevant here. > Hello, We had a similar issue last year with a group of users that created and removed files at a very high rate. Have you read https://docs.clyso.com/docs/kb/cephfs/#cephfs-pool-data-usage-growth-withou… ? We increased the purge rate parameters (very) aggressively to get back to a comfortable situation (i.e. not a pool w/ near full warnings). Loïc. -- | Loīc Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

CephFS large amounts of dark data
by Andrej Filipcic 09 Jun '26

09 Jun '26

Hi, I have a problem with large amounts of dark data/orphans on cephfs data pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd). There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3 EC. The layout for most files: ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216 pool=cephfs_data_echdd" But there is 1.4PiB discrepancy between the pool and the filesystem which I cannot explain and I suspect there are a lot of orphan objects there. I have run mds scrub on / and ~mdsdir as well. There is some mds damage on some old small files (~400 files), which I do not think it's relevant here. ceph was created with 17.2.6 and runs 19.2.3. Do you might have any suggestions how to fix this? Thanks, Andrej spod01 ~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 11 PiB 1.5 PiB 9.7 PiB 9.7 PiB 86.28 nvme 363 TiB 356 TiB 6.8 TiB 6.8 TiB 1.87 ssd 3.5 TiB 3.0 TiB 478 GiB 478 GiB 13.37 TOTAL 12 PiB 1.9 PiB 9.7 PiB 9.7 PiB 83.68 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 850 MiB 213 2.5 GiB 0 178 TiB cephfs_data 2 256 28 GiB 134.71M 86 GiB 2.86 972 GiB cephfs_metadata 3 64 66 GiB 6.66M 199 GiB 6.38 972 GiB cephfs_data_echdd 19 4096 7.0 PiB 594.70M 9.7 PiB 94.90 388 TiB cephfs_data_ecnvme 26 64 1.2 TiB 2.84M 1.7 TiB 0.48 251 TiB ... -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 0

Re: tentacle 20.2.2 QE validation status
by Nitzan Mordechai 09 Jun '26

09 Jun '26

Rados approved: https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2022-additional-PR-a… ________________________________ From: Yuri Weinstein via dev <dev(a)ceph.io> Sent: Monday, June 8, 2026 11:54 PM To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: tentacle 20.2.2 QE validation status We merged several PRs listed here https: //tracker. ceph. com/issues/77140 and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: Aviv, Vallari upgrade: We merged several PRs listed here https://tracker.ceph.com/issues/77140<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> and re-ran several suites. Seeking approvals: rados: Radek, Nitzan rgw: Adam Emerson fs: Venky orch/ceph-volume: Redo & Guillaume rbd: Ilya nvmeof: Aviv, Vallari upgrade: Patrick, Nitzan On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: Details of this release are summarized here: https://tracker.ceph.com/issues/75878#note-6<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade: Patrick upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 0

Re: tentacle 20.2.2 QE validation status
by Ilya Dryomov 09 Jun '26

09 Jun '26

On Mon, Jun 8, 2026 at 10:53 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > > We merged several PRs listed here https://tracker.ceph.com/issues/77140 and re-ran several suites. > Seeking approvals: > > rados: Radek, Nitzan > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redo & Guillaume > rbd: Ilya rbd approved. Thanks, Ilya

1 0

tentacle 20.2.2 QE validation status
by Yuri Weinstein 09 Jun '26

09 Jun '26

Details of this release are summarized here: https://tracker.ceph.com/issues/75878#note-6 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv, Vallari upgrade: Patrick upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 4

Re: Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 09 Jun '26

09 Jun '26

I've already tried that. No use. Cephadm has a problem with the hostname: mixtile@blade3n1:~$ sudo ceph cephadm osd activate blade3n1 Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: invalid literal for int() with base 10: 'blade3n1' I know that one or two people have had this error so far, but I have not found a remedy. Neither did cephadm deploy work: mixtile@blade3n1:~$ sudo cephadm deploy --osd-fsid 9f7fd40d-0698-40b9-8718-62942 b03e263 --name osd.blade3n1 --fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b --keyrin g /var/lib/ceph/8aad3073-39a1-11f1-bf6e-f2704a1efa9b/osd.blade3n1/keyring Deprecated command used: <function command_deploy at 0xffffa333cea0> Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.St atus}} ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b-osd-blade3n1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-8aad 3073-39a1-11f1-bf6e-f2704a1efa9b-osd-blade3n1 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.St atus}} ceph-8aad3073-39a1-11f1-bf6e-f2704a1efa9b-osd.blade3n1 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-8aad 3073-39a1-11f1-bf6e-f2704a1efa9b-osd.blade3n1 Deploy daemon osd.blade3n1 ... Shouldn't it create the necessary container itself? BTW, I've found out that the utilities ceph-osd, ceph-base, and ceph-volume were installed for some reason. I removed them, but that didn't help me either.

2 2

Re: MTU Change not going as planned
by Šarūnas Burdulis 09 Jun '26

09 Jun '26

On 6/8/26 7:58 AM, filip Mutterer via ceph-users wrote: > ... ... ... > auto eth0 > iface eth0 inet manual > pre-up ip link set dev eth0 mtu 9000 > > auto br0 > iface br0 inet static > address 192.168.1.41 > netmask 255.255.255.0 > gateway 192.168.1.1 > dns-nameservers 192.168.0.113 > bridge_ports eth0 > bridge_stp off > bridge_fd 0 > mtu 9000 > post-up ip link set dev br0 mtu 9000 > ... ... ... I haven't used ifconfig for a while now (using netplan), but us much as I remember, `mtu 9000` setting for br0 should suffice. I.e. no eth0 stanza and no pre/post ip link actions. As for OVN interference — why not temporarily disable it as a test? -- Šarūnas Burdulis Dartmouth Mathematics math.dartmouth.edu/~sarunas · https://useplaintext.email ·

1 0

CSC minutes 2026/06/08
by Redouane Kachach 09 Jun '26

09 Jun '26

Minutes from today's meeting of the Ceph Steering Committee: - *https://github.com/ceph/ceph/pull/68892 <https://github.com/ceph/ceph/pull/68892> PR to effect https://tracker.ceph.com/issues/76442 <https://tracker.ceph.com/issues/76442>* - The proposal is to continue the discussion on the PR - *Permissions to add/update "Labels" in redmine? Requires user to be admin/owner. Should PTLs be a part of those roles?* - The issue is custom fields concept in Redmine that can only be changed/added by a Redmine superuser (or admin) - Not a usual event (few times a year across components), just ask existing admins to create the labels for you - New Business - Debate on: - Motion: https://tracker.ceph.com/issues/76666 - Motion: https://tracker.ceph.com/issues/76648 - https://tracker.ceph.com/issues/76755 - https://tracker.ceph.com/issues/76682 - Election to vote on CSC member removals based on past (few) election participation - draft in https://tracker.ceph.com/issues/76665 - Election votes are not public (implementation of the process can be challenging) - Several concerns about whether that would be possible or not - *Can we fork OpenKMIP/PyKMIP: A Python implementation of the KMIP specification into Ceph GitHub? We need it for example in nvmeof gw, and it is currently unmaintained so we suggest to do it under ceph * - There's no community formed for those unmaintained projects. - Practical risks: security, no new Python versions support, etc - Used by SMB containers, nvmeof and probably by rgw - No clear forks with strong community - There was no immediate conclusion; concerns remain about what the fork implies - Simply having a fork doesn't imply a particular maintenance level - Need to revisit topic again later - *groups.io <http://groups.io> migration (?)* - Your Name on behalf of This List is a requirement - Not clear if this is a real requirement for mailing lists - No way disable receiving multiple copies sent to multiple lists - the alternative seems to be moving back the service to self-managed solution - Lab network policies can prevent from running a mail server on their infrastructure. * New Ceph Security Working Group approval * - Action item: Create a motion in Redmine

1 0

Ceph Tech Talk Reminder - Tuesday, June 9th, at 11 am PDT
by Anthony Middleton 08 Jun '26

08 Jun '26

Happy Monday, Ceph Friends! I hope you had a great weekend. Just a reminder that our next Ceph Tech Talk is tomorrow, *Tuesday, June 9th, at 11 am **PDT* on Building Ultra-Resilient Ceph Clusters with Xinnor and OSNexus. Steven Umbehocker, CEO of OSNexus, and Dmitry Livshits, CEO of Xinnor, will lead the session. Together we will cover: - Multi-site durability challenges - Compound failure scenarios - Transparent RAID beneath Ceph OSDs - Operational simplicity with native integration *Here is the link: https://meet.jit.si/ceph-tech-talks <https://meet.jit.si/ceph-tech-talks>* If you have suggestions for a future session, please let me know. My email address is below. 😉 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

MTU Change not going as planned
by filip Mutterer 08 Jun '26

08 Jun '26

Hi! I am trying to set the MTU to 9000. ip link set dev br0 mtu 9000 It does work after running the command. ip link show br0 4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 22:6c:63:46:7e:67 brd ff:ff:ff:ff:ff:ff But adding it to my configuration like this: auto lo iface lo inet loopback auto eth0 iface eth0 inet manual pre-up ip link set dev eth0 mtu 9000 auto br0 iface br0 inet static address 192.168.1.41 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 192.168.0.113 bridge_ports eth0 bridge_stp off bridge_fd 0 mtu 9000 post-up ip link set dev br0 mtu 9000 The result after a restart is eth0 got MTU 9000 but br0 sticks to 1500. I can change it and seems to work fine. Tested it like that: ping -M do -s 8950 192.168.1.42 Also I noticed runing service networking restart && ip link show br0 Shows the MTu 9000 is set once. I also have an working OVN and OpenVSwitch on that machines. Is it OVN/OVS that makes it comming out like that? Greetings filip

1 0

Re: tentacle 20.2.2 QE validation status
by Guillaume ABRIOUX 08 Jun '26

08 Jun '26

Hi Yuri, There are still failures like: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-06-01_16:46:10-orch-tent… : could be fixed with [1] but it is not going to be included in 20.2.2 as the milestone was replaced by 20.2.3, see [2] https://qa-proxy.ceph.com/teuthology/pdonnell-2026-06-01_16:46:10-orch-tent… : seems to be a valid issue in orchestrator (cephadm doesn’t actually stop OSD deamons before making ceph-volume zap the underlying device) From ceph-volume perspective, the other failures I see are unrelated or due to infra/job definition issues. Thanks! [1] https://github.com/ceph/ceph/pull/68586/changes/b9a0dcc7da9196bf4d505dfe6dd… [2] https://github.com/ceph/ceph/pull/68586#event-25553632788 -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Thursday, 4 June 2026 at 22:44 To: Yuri Weinstein <yweinste(a)redhat.com>; Redouane Kachach <rkachach(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: tentacle 20.2.2 QE validation status My apologies typo for orch dev lead name: I am still seeking approvals for: fs: Venky orch/ceph-volume: Redo & Guillaume On Thu, Jun 4, 2026 at 1:37 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > I am still seeking approvals for: > > fs: Venky > orch/ceph-volume: Adam King & Guillaume > > We have an outstanding PRs tracker https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… , > that need to be approved and PRs merged before we can finish testing > this release. > > We also plan for the old LRC to be upgraded to use this release. > > TIA > > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA >> > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Dan van der Ster 06 Jun '26

06 Jun '26

On Fri, Jun 5, 2026 at 1:05 AM Iztok Gregori via ceph-users <ceph-users(a)ceph.io> wrote: > Just to be sure for the EMEA event, is the time CET (UTC+1) and not CEST > (UTC+2)? Doh! You're right -- CEST and PDT. The eventbrite pages have been updated.

1 0

Re: Best "modern" approach to drain/fill a OSD server
by Alvaro Soto 06 Jun '26

06 Jun '26

Hi Dan, Can I share this with the LATAM community? Cheers! On Thu, Jun 4, 2026 at 4:09 PM Dan van der Ster via ceph-users < ceph-users(a)ceph.io> wrote: > Hi all, > > We'll host a couple of OSD Balancing Happy Hours next week. Sign up here to > join: > > EMEA friendly -- 5:00-6:00 PM CET -- > https://www.eventbrite.com/e/1990653154171 > > NA friendly -- 5:00-6:00 PM PT -- > https://www.eventbrite.com/e/1990658333663 > > Hope to see you there, > Dan > > On Tue, Apr 28, 2026 at 6:50 AM Janne Johansson <icepic.dz(a)gmail.com> > wrote: > > > > If it can be useful, I'm thinking of organizing an OSD balancing happy > > > hour call in the next month or so. > > > Maybe mid-May or so on a Friday afternoon (EU time) / morning (US > time). > > > > Also interested. > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

2 1

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 05 Jun '26

05 Jun '26

On Wed, Jun 3, 2026 at 12:42 PM Eugen Block via dev <dev(a)ceph.io> wrote: > > Now that's unexpected, I thought I had misremembered my previous > actions and didn't mention it yet, now I created 3 new VMs, created a > new group, took a group snapshot, all good. But a group snap rollback > is executed although the rbd images have watchers: > > rbd --id openstack group snap create images/test-servers@snap1 > > rbd --id openstack group image ls images/test-servers > images/3df08789-2be9-4e99-9746-9d2edc8c612a_disk > images/7a5d19eb-1034-489f-885a-0074fef59e89_disk > images/f79e323f-87a1-4cb7-ad9c-1108ce73efe3_disk > > > rbd status images/3df08789-2be9-4e99-9746-9d2edc8c612a_disk > Watchers: > watcher=X.X.X.18:0/4236769191 client.<client> cookie=<cookie> > > rbd --id openstack group snap rollback images/test-servers@snap1 > Rolling back to group snapshot: 100% complete...done. > > This shouldn't be possible, I would expect a message like this: > > Rolling back to snapshot: 0% complete...failed. > rbd: rollback failed: (30) Read-only file system > > Is this a known bug? No, this behavior is expected. The reason for why "rbd snap rollback" command can deny the operation with EROFS when the image is mapped in some cases but "rbd group snap rollback" command doesn't do that lies in implementation details (specifically the interaction with exclusive locks on member images). When it comes to rollback operation, the mere presence of a watch isn't really an indicator of anything. That said, I'd recommend shutting down clients before issuing any rollbacks in both standalone image and group scenarios. That way all caches would get invalidated and there is no chance of confusing someone or something with now-stale data. Thanks, Ilya > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > But one more question on this: why is it allowed to remove an image > > from the group if there are existing snapshots? Shouldn't this be > > prevented to keep the group consistency? > > > > And just for my understanding: how are the group snapshots > > technically created? Is that one snapshot for all images or is it an > > individual snapshot per image? > > > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > >> I understand, you're right about the group consistency of course. I > >> just thought if you can remove an image from the group, it would > >> also remove the image's snapshot(s) from the list of snapshots as > >> well. My scenario is: initially I thought it would make sense to > >> have those servers in a group because if I wanted to rollback, it > >> would make sense to do it for all. But then I thought a bit more > >> about it and decided that one of the images actually doesn't make > >> sense to be in that group. Re-adding it will cause more problems in > >> case of rollback... I need to think about this... > >> > >> Thanks a lot for taking the time, I really appreciate it! > >> > >> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >> > >>> On Wed, Jun 3, 2026 at 10:27 AM Eugen Block <eblock(a)nde.ag> wrote: > >>>> > >>>> Hi, > >>>> > >>>> that is correct, log_to_stderr is false in our cluster. And with > >>>> --log-to-stderr true the result is a you expected: > >>>> > >>>> rbd: rollback group to snapshot failed: 2026-06-03T08:06:52.930+0000 > >>>> 7f53896ae0c0 -1 librbd::api::Group: snap_rollback: group snapshot > >>>> membership does not match group membership > >>>> > >>>> But what's the conclusion here? So it's not allowed to rollback if > >>>> memberships don't match. How would I correct the membership? > >>> > >>> Re-add the image back to the group if the image is still around. > >>> > >>>> Because I > >>>> wouldn't want to delete all snapshots from before I removed one image > >>>> from the group. Is there any workaround? > >>> > >>> I'm not sure I see what needs to be worked around here. The group is > >>> supposed to be a logical collection of images where some level of > >>> consistency between images is required, not a random "bag". This > >>> suggests that while images can come and go (i.e. be added and removed > >>> from the group), the group can't always be meaningfully rolled back. > >>> For example, if a group snapshot captured images A, B and C but image > >>> C had since been removed from the group and potentially reformatted, > >>> repurposed for something else or removed altogether, the group's state > >>> exactly as of that snapshot just can't be restored. > >>> > >>> Thanks, > >>> > >>> Ilya > >>> > >>>> > >>>> Thanks, > >>>> Eugen > >>>> > >>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>> > >>>>> On Fri, May 29, 2026 at 6:41 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>>> > >>>>>> The commands were: > >>>>>> > >>>>>> controller02:~# rbd --id user group create images/test-servers > >>>>>> > >>>>>> controller02:~# for i in 0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd > >>>>>> 72f5816c-c1db-44de-b0a2-19d661faa963 > >>>>>> 47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc; do rbd --id user group image add > >>>>>> images/test-servers images/${i}_disk; done > >>>>>> > >>>>>> controller02:~# rbd --id user group image ls images/test-servers > >>>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>>> images/47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc_disk > >>>>>> images/72f5816c-c1db-44de-b0a2-19d661faa963_disk > >>>>>> > >>>>>> controller02:~# rbd --id user group snap create > >>>>>> images/test-servers@snap1 > >>>>>> > >>>>>> controller02:~# rbd --id user group snap ls images/test-servers > >>>>>> NAME STATUS > >>>>>> snap1 ok > >>>>>> > >>>>>> > >>>>>> # rollback works for all images > >>>>>> controller02:~# rbd --id user group snap rollback > >>>>>> images/test-servers@snap1 > >>>>>> Rolling back to group snapshot: 100% complete...done. > >>>>>> > >>>>>> # removing one image from the group > >>>>>> controller02:~# rbd --id user group image rm images/test-servers > >>>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>>> > >>>>>> # rollback fails > >>>>>> controller02:~# rbd --id user group snap rollback > >>>>>> images/test-servers@snap1 > >>>>>> Rolling back to group snapshot: 0% complete...failed. > >>>>>> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>>> > >>>>>> I'll add the debug output later, will need to sanitze it first. But I > >>>>>> don't see anything obvious in there. > >>>>> > >>>>> Hi Eugen, > >>>>> > >>>>> Based on the above, it's https://tracker.ceph.com/issues/66300 and is > >>>>> therefore the intended behavior. The only fly in the ointment is that > >>>>> you aren't seeing the associated "group snapshot membership does not > >>>>> match group membership" error message. > >>>>> > >>>>> You not seeing it is consistent with the attached debug output where > >>>>> only very early messenger traffic is present and nothing beyond that. > >>>>> It suggests some non-conventional settings in the cluster-wide config > >>>>> such as log_to_stderr being set to false or similar. > >>>>> > >>>>> Can you try appending --log-to-stderr true to "rbd group snap rollback" > >>>>> command? > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Ilya > >>>>> > >>>>>> > >>>>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>>> > >>>>>>> On Fri, May 29, 2026 at 4:05 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> thanks for your quick reply. No I didn't see any additional output > >>>>>>>> than the one I shared (invalid argument). I could add debug log level > >>>>>>>> if necessary. > >>>>>>> > >>>>>>> That error message should have been displayed no matter the log level, > >>>>>>> so something other than https://tracker.ceph.com/issues/66300 might be > >>>>>>> involved. > >>>>>>> > >>>>>>> What exactly do you mean by "I removed an image from the group > >>>>>>> snapshot"? Which commands were run there and in what order? > >>>>>>> > >>>>>>>> But one more detail, I also tried the rollback directly within the > >>>>>>>> cephadm shell (so version 19.2.3) with the same result: > >>>>>>>> > >>>>>>>> ceph03:~ # cephadm shell > >>>>>>>> ... > >>>>>>>> [ceph: root@ceph03 /]# rbd group snap rollback > >>>>>>>> images/test-servers@20260430_start > >>>>>>>> Rolling back to group snapshot: 0% complete...failed. > >>>>>>>> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>>>>> > >>>>>>>> [ceph: root@ceph03 /]# ceph -v > >>>>>>>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > >>>>>>>> squid (stable) > >>>>>>> > >>>>>>> Can you try appending --debug-ms 1 --debug-rbd 20 to the command > >>>>>>> (let's stick to this cephadm shell) and attach the output? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Ilya > >>>>>>> > >>>>>>>> > >>>>>>>> Thanks! > >>>>>>>> Eugen > >>>>>>>> > >>>>>>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>>>>> > >>>>>>>> > On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users > >>>>>>>> > <ceph-users(a)ceph.io> wrote: > >>>>>>>> >> > >>>>>>>> >> Hi, > >>>>>>>> >> > >>>>>>>> >> I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > >>>>>>>> >> version 19.2.1), the Ceph cluster version is 19.2.3. The > >>>>>>>> client fails > >>>>>>>> >> with "invalid argument": > >>>>>>>> >> > >>>>>>>> >> controller02:~# rbd --id <user> group snap rollback --pool images > >>>>>>>> >> --group test-servers --snap 20260430_start > >>>>>>>> >> Rolling back to group snapshot: 0% complete...failed. > >>>>>>>> >> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>>>>> >> > >>>>>>>> >> controller02:~# ceph -v > >>>>>>>> >> ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) > >>>>>>>> >> squid (stable) > >>>>>>>> >> > >>>>>>>> >> But running the same command (just as admin not as <user>) > >>>>>>>> on a Ceph > >>>>>>>> >> node works: > >>>>>>>> >> > >>>>>>>> >> ceph03:~ # rbd group snap rollback --pool images --group > >>>>>>>> test-servers > >>>>>>>> >> --snap 20260430_start > >>>>>>>> >> Rolling back to group snapshot: 100% complete...done. > >>>>>>>> >> > >>>>>>>> >> ceph03:~ # ceph -v > >>>>>>>> >> ceph version 16.2.13-66-g54799ee0666 > >>>>>>>> >> (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> What seems to be the issue here is that I removed an image from the > >>>>>>>> >> group snapshot. I wonder if it could be this bug [0] which > >>>>>>>> is supposed > >>>>>>>> >> to be fixed in 19.2.0 according to the "Released In" field of the > >>>>>>>> >> Squid backport tracker [1]. > >>>>>>>> >> > >>>>>>>> >> This seems a little inconsistent to me, could someone > >>>>>>>> please clarify? > >>>>>>>> > > >>>>>>>> > Hi Eugen, > >>>>>>>> > > >>>>>>>> > Did you see "group snapshot membership does not match group > >>>>>>>> membership" > >>>>>>>> > error message when the rollback command failed with 19.2.1 client? > >>>>>>>> > > >>>>>>>> > Thanks, > >>>>>>>> > > >>>>>>>> > Ilya > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >>>> > >

2 3

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 05 Jun '26

05 Jun '26

> If it can be useful, I'm thinking of organizing an OSD balancing happy > hour call in the next month or so. > Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Also interested. -- May the most significant bit of your life be positive.

3 2

Re: [EXT] Crash on newly created NVMe+HDD OSD
by Justin Mammarella 05 Jun '26

05 Jun '26

Hi Jan, We encountered this issue after upgrading to Tentacle 20.2.1 and attempting to rebuild some of our OSDs that use an NVMe device for the WAL/DB. I was unable to find any instance of this issue being reported upstream, either via a GitHub pull request or a bug report. My colleague has submitted a bug report here with the code regression, including a workaround: https://tracker.ceph.com/issues/77127 Justin Mammarella Storage Infrastructure Lead Research Computing Services The University of Melbourne Victoria 3010, Australia From: Jan Kasprzak via ceph-users <ceph-users(a)ceph.io> Date: Monday, 30 March 2026 at 6:17 pm To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [EXT] [ceph-users] Crash on newly created NVMe+HDD OSD External email: Please exercise caution Hello, Ceph users, I wanted to replace a HDD with bad blocks in an OSD which has block.db on NVMe. Removing the OSD (ceph-volume lvm zap, etc.) went well, no VG/LV remained for data or metadata. But after replacing the HDD, preparing a new HDD+NVMe OSD, and starting it with "ceph-volume lvm activate --all", the OSD daemon crashed with terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input Retrying the proces once more I noticed that ceph-volume created a VG/PV/LV on the HDD (/dev/sda), but did not create a VG/PV/LV on /dev/nvme1n1p4. Zapping the result once more and creating a HDD-only OSD helped, the OSD daemon started and joined the cluster correctly. I then stopped it once more, and tried to create a db volume myself, and migrate db to it: systemctl stop ceph-osd@$OSD_ID NEW_DB_VG=ceph-`uuidgen` NEW_DB_LV=osd-db-`uuidgen` vgcreate $NEW_DB_VG /dev/nvme1n1p4 lvcreate -a y -l 100%FREE -n $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm new-db --osd-id $OSD_ID --osd-fsid $OSD_FSID --target $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm migrate --osd-id $OSD_ID --osd-fsid $OSD_FSID --from data --target $NEW_DB_VG/$NEW_DB_LV systemctl start ceph-osd@$OSD_ID ... and it started correctly and did not crash. I used the same commands previously on Ceph 19 when replacing HDDs, and it worked without problem. This is the first time I replaced a HDD after upgrading to Ceph 20.2.0, so it might be a regression in ceph-volume of 20.2.0. Below is the output of "ceph-volume lvm prepare", and the crash report from journal. Did I do something incorrect here? Thanks! -Yenya # ceph-volume lvm prepare --bluestore --block.db /dev/nvme1n1p4 --data /dev/sda Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: vgcreate --force --yes ceph-8805a195-57fd-4054-8282-a307d15955a5 /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-8805a195-57fd-4054-8282-a307d15955a5" successfully created Running command: lvcreate --yes -l 2861055 -n osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 ceph-8805a195-57fd-4054-8282-a307d15955a5 stdout: Logical volume "osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963" created. Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-25 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 Running command: /usr/bin/ln -s /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 /var/lib/ceph/osd/ceph-25/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-25/activate.monmap stderr: got monmap epoch 39 --> Creating keyring file for osd.25 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/keyring Running command: /usr/bin/chown -R ceph:ceph /dev/nvme1n1p4 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 25 --monmap /var/lib/ceph/osd/ceph-25/activate.monmap --keyfile - --bluestore-block-db-path /dev/nvme1n1p4 --osd-data /var/lib/ceph/osd/ceph-25/ --osd-uuid db38dbfb-e161-4680-93e3-ba4abaa36963 --setuser ceph --setgroup ceph stderr: 2026-03-30T07:59:06.150+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25//block) No valid bdev label found stderr: 2026-03-30T07:59:06.175+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/sda Mar 30 07:52:31 osdhost systemd[1]: Starting Ceph object storage daemon osd.25... Mar 30 07:52:31 osdhost systemd[1]: Started Ceph object storage daemon osd.25. Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' Mar 30 07:52:31 osdhost ceph-osd[1140285]: what(): static void bluefs_fnode_t::_denc_finish(ceph::buffer::v15_2_0::ptr::const_iterator&, char**, uint32_t*): Malformed input [buffer:3] Mar 30 07:52:31 osdhost ceph-osd[1140285]: *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Main process exited, code=dumped, status=6/ABRT Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Failed with result 'core-dump'. -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: coe, an adventure story
by George Shuklin 05 Jun '26

05 Jun '26

On 6/4/26 1:33 AM, C U via ceph-users wrote: > Enjoy! Lessons learned and demonstration of "AI enhanced" infrastructure. > > TL:DR used ceph-kvstore-tool bluestore-k work around a data corruption > issue cause by machine operator. > > https://github.com/tu503/coe/blob/main/2026-06-03-ceph-bulk-rbd-rm/COE.md This is the fine piece of reading, thank you. There are few things which are not clear to me: 1. If osd fails to start after abort, this is a bug. There is no link to issue anywhere in the report. Was bug identified, or at least reported? 2. It says that controller was overloaded. At the same time, there was a notice on hard drives behind controller. Are you sure it was controller overloaded, and not a normal drive thrashing? Was you able to reproduce this with synthetic load? Was was ratio between drives and ports (and of what kind?) in your system?

1 0

Re: tentacle 20.2.2 QE validation status
by Yuri Weinstein 05 Jun '26

05 Jun '26

My apologies typo for orch dev lead name: I am still seeking approvals for: fs: Venky orch/ceph-volume: Redo & Guillaume On Thu, Jun 4, 2026 at 1:37 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > I am still seeking approvals for: > > fs: Venky > orch/ceph-volume: Adam King & Guillaume > > We have an outstanding PRs tracker https://tracker.ceph.com/issues/77086, > that need to be approved and PRs merged before we can finish testing > this release. > > We also plan for the old LRC to be upgraded to use this release. > > TIA > > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75878#note-6 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA >> >

1 0

coe, an adventure story
by C U 04 Jun '26

04 Jun '26

Enjoy! Lessons learned and demonstration of "AI enhanced" infrastructure. TL:DR used ceph-kvstore-tool bluestore-k work around a data corruption issue cause by machine operator. https://github.com/tu503/coe/blob/main/2026-06-03-ceph-bulk-rbd-rm/COE.md regards,

1 0

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 03 Jun '26

03 Jun '26

On Wed, Jun 3, 2026 at 12:01 PM Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > But one more question on this: why is it allowed to remove an image > from the group if there are existing snapshots? Shouldn't this be > prevented to keep the group consistency? It wasn't prevented when groups were originally introduced and was considered to be too restrictive for a retroactive fix. Removing an image from the group doesn't invalidate any group snapshots -- the corresponding image snapshots remain there. An image can always be re-added to the group and as long as the operator is OK with the HEAD of the image getting rolled back after the image was potentially used for something else while outside the group, "rbd group snap rollback" command can be run and would work as expected. > > And just for my understanding: how are the group snapshots technically > created? Is that one snapshot for all images or is it an individual > snapshot per image? The latter -- an individual snapshot per member image. You can see these snapshots if you pass --all to "rbd snap ls" command. Thanks, Ilya > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > I understand, you're right about the group consistency of course. I > > just thought if you can remove an image from the group, it would > > also remove the image's snapshot(s) from the list of snapshots as > > well. My scenario is: initially I thought it would make sense to > > have those servers in a group because if I wanted to rollback, it > > would make sense to do it for all. But then I thought a bit more > > about it and decided that one of the images actually doesn't make > > sense to be in that group. Re-adding it will cause more problems in > > case of rollback... I need to think about this... > > > > Thanks a lot for taking the time, I really appreciate it! > > > > Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > > > >> On Wed, Jun 3, 2026 at 10:27 AM Eugen Block <eblock(a)nde.ag> wrote: > >>> > >>> Hi, > >>> > >>> that is correct, log_to_stderr is false in our cluster. And with > >>> --log-to-stderr true the result is a you expected: > >>> > >>> rbd: rollback group to snapshot failed: 2026-06-03T08:06:52.930+0000 > >>> 7f53896ae0c0 -1 librbd::api::Group: snap_rollback: group snapshot > >>> membership does not match group membership > >>> > >>> But what's the conclusion here? So it's not allowed to rollback if > >>> memberships don't match. How would I correct the membership? > >> > >> Re-add the image back to the group if the image is still around. > >> > >>> Because I > >>> wouldn't want to delete all snapshots from before I removed one image > >>> from the group. Is there any workaround? > >> > >> I'm not sure I see what needs to be worked around here. The group is > >> supposed to be a logical collection of images where some level of > >> consistency between images is required, not a random "bag". This > >> suggests that while images can come and go (i.e. be added and removed > >> from the group), the group can't always be meaningfully rolled back. > >> For example, if a group snapshot captured images A, B and C but image > >> C had since been removed from the group and potentially reformatted, > >> repurposed for something else or removed altogether, the group's state > >> exactly as of that snapshot just can't be restored. > >> > >> Thanks, > >> > >> Ilya > >> > >>> > >>> Thanks, > >>> Eugen > >>> > >>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>> > >>>> On Fri, May 29, 2026 at 6:41 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>> > >>>>> The commands were: > >>>>> > >>>>> controller02:~# rbd --id user group create images/test-servers > >>>>> > >>>>> controller02:~# for i in 0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd > >>>>> 72f5816c-c1db-44de-b0a2-19d661faa963 > >>>>> 47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc; do rbd --id user group image add > >>>>> images/test-servers images/${i}_disk; done > >>>>> > >>>>> controller02:~# rbd --id user group image ls images/test-servers > >>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>> images/47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc_disk > >>>>> images/72f5816c-c1db-44de-b0a2-19d661faa963_disk > >>>>> > >>>>> controller02:~# rbd --id user group snap create images/test-servers@snap1 > >>>>> > >>>>> controller02:~# rbd --id user group snap ls images/test-servers > >>>>> NAME STATUS > >>>>> snap1 ok > >>>>> > >>>>> > >>>>> # rollback works for all images > >>>>> controller02:~# rbd --id user group snap rollback > >>>>> images/test-servers@snap1 > >>>>> Rolling back to group snapshot: 100% complete...done. > >>>>> > >>>>> # removing one image from the group > >>>>> controller02:~# rbd --id user group image rm images/test-servers > >>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>> > >>>>> # rollback fails > >>>>> controller02:~# rbd --id user group snap rollback > >>>>> images/test-servers@snap1 > >>>>> Rolling back to group snapshot: 0% complete...failed. > >>>>> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>> > >>>>> I'll add the debug output later, will need to sanitze it first. But I > >>>>> don't see anything obvious in there. > >>>> > >>>> Hi Eugen, > >>>> > >>>> Based on the above, it's https://tracker.ceph.com/issues/66300 and is > >>>> therefore the intended behavior. The only fly in the ointment is that > >>>> you aren't seeing the associated "group snapshot membership does not > >>>> match group membership" error message. > >>>> > >>>> You not seeing it is consistent with the attached debug output where > >>>> only very early messenger traffic is present and nothing beyond that. > >>>> It suggests some non-conventional settings in the cluster-wide config > >>>> such as log_to_stderr being set to false or similar. > >>>> > >>>> Can you try appending --log-to-stderr true to "rbd group snap rollback" > >>>> command? > >>>> > >>>> Thanks, > >>>> > >>>> Ilya > >>>> > >>>>> > >>>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>> > >>>>> > On Fri, May 29, 2026 at 4:05 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>> >> > >>>>> >> Hi, > >>>>> >> > >>>>> >> thanks for your quick reply. No I didn't see any additional output > >>>>> >> than the one I shared (invalid argument). I could add debug log level > >>>>> >> if necessary. > >>>>> > > >>>>> > That error message should have been displayed no matter the log level, > >>>>> > so something other than https://tracker.ceph.com/issues/66300 might be > >>>>> > involved. > >>>>> > > >>>>> > What exactly do you mean by "I removed an image from the group > >>>>> > snapshot"? Which commands were run there and in what order? > >>>>> > > >>>>> >> But one more detail, I also tried the rollback directly within the > >>>>> >> cephadm shell (so version 19.2.3) with the same result: > >>>>> >> > >>>>> >> ceph03:~ # cephadm shell > >>>>> >> ... > >>>>> >> [ceph: root@ceph03 /]# rbd group snap rollback > >>>>> >> images/test-servers@20260430_start > >>>>> >> Rolling back to group snapshot: 0% complete...failed. > >>>>> >> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>> >> > >>>>> >> [ceph: root@ceph03 /]# ceph -v > >>>>> >> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > >>>>> >> squid (stable) > >>>>> > > >>>>> > Can you try appending --debug-ms 1 --debug-rbd 20 to the command > >>>>> > (let's stick to this cephadm shell) and attach the output? > >>>>> > > >>>>> > Thanks, > >>>>> > > >>>>> > Ilya > >>>>> > > >>>>> >> > >>>>> >> Thanks! > >>>>> >> Eugen > >>>>> >> > >>>>> >> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>> >> > >>>>> >> > On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users > >>>>> >> > <ceph-users(a)ceph.io> wrote: > >>>>> >> >> > >>>>> >> >> Hi, > >>>>> >> >> > >>>>> >> >> I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > >>>>> >> >> version 19.2.1), the Ceph cluster version is 19.2.3. The > >>>>> client fails > >>>>> >> >> with "invalid argument": > >>>>> >> >> > >>>>> >> >> controller02:~# rbd --id <user> group snap rollback --pool images > >>>>> >> >> --group test-servers --snap 20260430_start > >>>>> >> >> Rolling back to group snapshot: 0% complete...failed. > >>>>> >> >> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>> >> >> > >>>>> >> >> controller02:~# ceph -v > >>>>> >> >> ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) > >>>>> >> >> squid (stable) > >>>>> >> >> > >>>>> >> >> But running the same command (just as admin not as <user>) > >>>>> on a Ceph > >>>>> >> >> node works: > >>>>> >> >> > >>>>> >> >> ceph03:~ # rbd group snap rollback --pool images --group > >>>>> test-servers > >>>>> >> >> --snap 20260430_start > >>>>> >> >> Rolling back to group snapshot: 100% complete...done. > >>>>> >> >> > >>>>> >> >> ceph03:~ # ceph -v > >>>>> >> >> ceph version 16.2.13-66-g54799ee0666 > >>>>> >> >> (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > >>>>> >> >> > >>>>> >> >> > >>>>> >> >> What seems to be the issue here is that I removed an image from the > >>>>> >> >> group snapshot. I wonder if it could be this bug [0] which > >>>>> is supposed > >>>>> >> >> to be fixed in 19.2.0 according to the "Released In" field of the > >>>>> >> >> Squid backport tracker [1]. > >>>>> >> >> > >>>>> >> >> This seems a little inconsistent to me, could someone > >>>>> please clarify? > >>>>> >> > > >>>>> >> > Hi Eugen, > >>>>> >> > > >>>>> >> > Did you see "group snapshot membership does not match group > >>>>> membership" > >>>>> >> > error message when the rollback command failed with 19.2.1 client? > >>>>> >> > > >>>>> >> > Thanks, > >>>>> >> > > >>>>> >> > Ilya > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> > >>>>> > >>>>> > >>> > >>> > >>> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 03 Jun '26

03 Jun '26

On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > version 19.2.1), the Ceph cluster version is 19.2.3. The client fails > with "invalid argument": > > controller02:~# rbd --id <user> group snap rollback --pool images > --group test-servers --snap 20260430_start > Rolling back to group snapshot: 0% complete...failed. > rbd: rollback group to snapshot failed: (22) Invalid argument > > controller02:~# ceph -v > ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) squid (stable) > > But running the same command (just as admin not as <user>) on a Ceph > node works: > > ceph03:~ # rbd group snap rollback --pool images --group test-servers > --snap 20260430_start > Rolling back to group snapshot: 100% complete...done. > > ceph03:~ # ceph -v > ceph version 16.2.13-66-g54799ee0666 > (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > > > What seems to be the issue here is that I removed an image from the > group snapshot. I wonder if it could be this bug [0] which is supposed > to be fixed in 19.2.0 according to the "Released In" field of the > Squid backport tracker [1]. > > This seems a little inconsistent to me, could someone please clarify? Hi Eugen, Did you see "group snapshot membership does not match group membership" error message when the rollback command failed with 19.2.1 client? Thanks, Ilya

2 10

Re: The mail threading on this list is broken
by Frédéric Nass 03 Jun '26

03 Jun '26

That's great news! Thank you, Adam, for letting everyone know. We're looking forward to it. Best regards, Frédéric -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass(a)clyso.com Le dim. 31 mai 2026 à 16:22, Adam Kraitman via ceph-users < ceph-users(a)ceph.io> a écrit : > Thank you for the detailed analysis, the threading issue you've described > is a known problem with the current mailing list infrastructure. We're > planning to migrate all mailing lists to Groups.io, and this issue will be > resolved as part of that migration. > > We'll share more details and the migration timeline once everything is > finalized > > On Sun, May 31, 2026 at 3:08 PM Kai Stian Olstad via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Hi > > > > For several month the mail threading on this list broken and making the > > list hard to read. > > It can easily be seen by looking at the archive, in this link for about > > 9 mount(as of writing) > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > > > > Subjects all have nice subject without "Re:" for replay, and on the > > right you see how may people(person icon) in the thread and number of > > post(talking bubble) in the thread. > > > > But if we look at more recent posts like this one > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > > you see a lot of "Re:" in the subject and most of them have one person > > and only one post because the threading is broken. > > > > Take this recent mail "Weird behavior for 2 OSDs in our cluster" > > The fist mail by Wannes Smet > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/62B63BH33F3… > > when i check my mail client this mail has following ID. > > "Message-ID: > > < > > > 0100019e7c77595f-ffb4d04b-1771-46cf-8b60-10795bc6a1c5-000000(a)email.amazonses.com > > >" > > > > The reply from Eugen Block > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/E63Z3JR7DMS… > > the header says this is a reply to different ID > > "In-Reply-To: > > < > > > 0100019e7c7979e5-8d4d96e6-1c93-4a76-a5cd-efdf25ada853-000000(a)email.amazonses.com > > >" > > > > When I search the header of the mail from Wannes Smet I find no such ID > > that is in the In-Reply-To. > > > > It looks to me that the mailing-list server is sending different > > "Messages-ID" to the members on the list. > > This should be the same or the threading breaks as we can se. > > > > Can this be fixed please? > > > > -- > > Kai Stian Olstad > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

test requested y Anthony Middleton
by zouhair nador 03 Jun '26

03 Jun '26

Hello [cid:d40f5272-f460-403e-be1b-ca2561377aed]

1 1

CFP: PDSW 2026 @ SC26 - Submission Deadline July 31, 2026
by Neuwirth, Prof. Dr. Sarah 03 Jun '26

03 Jun '26

-------------------------------------------------------------------------- The 11th International Parallel Data Systems Workshop (PDSW'26) -------------------------------------------------------------------------- PDSW 2026 website: https://www.pdsw.org/ Paper Submissions due: July 31st, 2026, 11:59 PM AoE AD due: August 7th, 2026, 11:59 PM AoE Paper Notification: Sep 4th, 2026, 11:59 PM AoE Camera ready due: Sep 25th, 2026, 11:59 PM AoE Final AD/AE due: Sep 25th, 2026, 11:59 PM AoE Workshop day: Monday, Nov 16th, 2026 (all day) Submissions website: https://submissions.supercomputing.org/ We are excited to announce the 11th International Parallel Data Systems Workshop (PDSW'26), to be held in conjunction with SC26: The International Conference for High Performance Computing, Networking, Storage, and Analysis, in Chicago, IL. PDSW'26 builds upon the rich legacy of its predecessor workshops, the Petascale Data Storage Workshop (PDSW, 2006–2015) and the Data Intensive Scalable Computing Systems (DISCS, 2012–2015) workshop. The increasing importance of efficient data storage and management continues to drive scientific productivity across traditional simulation-based HPC environments and emerging Cloud, AI/ML, and Big Data analysis frameworks. Challenges are compounded by the rapidly expanding volumes of experimental and observational data, the growing disparity between computational and storage hardware performance, and the rise of novel data-driven algorithms in machine learning. This workshop aims to advance research and development by addressing the most pressing challenges in large-scale data storage and processing. We invite the community to contribute original research manuscripts that introduce and evaluate novel algorithms or architectures, share significant scientific case studies or workloads, or assess the reproducibility of previously published work. We emphasize the importance of community collaboration for problem identification, workload capture, solution interoperability, standardization, and shared tools. Authors are encouraged to provide comprehensive experimental environment details (software versions, benchmark configurations, etc.) to promote transparency and facilitate collaborative progress. ----------------------------- Topics of Interest ----------------------------- - Scalable Architectures: Distributed data storage, archival, and virtualization. - New Data Processing Models and Algorithms: Application of innovative data processing models and algorithms for parallel computing and analysis. - Performance Analysis: Benchmarking, resource management, and workload studies. - Cloud and Container-Based Models: Enabling cloud and container-based frameworks for large-scale data analysis. - Storage Technologies: Adaptation to emerging hardware and computing models. - Data Integrity: Techniques to ensure data integrity, availability, reliability, and fault tolerance. - Programming Models and Frameworks: Big data solutions for data-intensive computing. - Hybrid Cloud Data Processing: Integration of hybrid cloud and on-premise data processing. - Cloud-Specific Opportunities: Data storage and transit opportunities specific to cloud computing. - Storage System Programmability: Enhancing programmability in storage systems. - Data Reduction Techniques: Filtering, compression, and reduction techniques for large-scale data. - File and Metadata Management: Parallel file systems, metadata management at scale. - In-Situ and In-Transit Processing: Integrating computation into the memory and storage hierarchy for in-situ and in-transit data processing. - Alternative Storage Models: Object stores, key-value stores, and other data storage models. - Productivity Tools: Tools for data-intensive computing, data mining, and knowledge discovery. - Data Movement: Managing data movement between compute and data-intensive components. - Cross-Cloud Data Management: Efficient data management across different cloud environments. - AI-enhanced Systems: Storage system optimization and data analytics using machine learning. - New Memory and Storage Systems: Innovative techniques and performance evaluation for new memory and storage systems. - AI and Agentic related data management: tools and techniques necessary to support AI workloads and Agentic AI data analytics for online decision making. More details are available at: https://www.pdsw.org/ ----------------------------- Template and Submission ----------------------------- - A full paper up to 6 pages in length, excluding references, acknowledgements, and AD/AE appendices. Once anonymization is removed for camera ready submission, these limits must still be met. Please make sure that author names fit within the 6-page limit as well even though the submission should be double-anonymized. - Artifact Description (AD) Appendix is mandatory and Artifact Evaluation (AE) Appendix is optional. - Papers must adhere to the IEEE conference proceeding template available at: https://www.ieee.org/conferences/publishing/templates - Papers will be reviewed double-blind. Author names and affiliations should NOT be included in the submitted paper. - Submit your papers by July 31st, 2026, 11:59 PM AoE at https://submissions.supercomputing.org/ ----------------------------- Reproducibility Initiative ----------------------------- Aligned with the SC26 Reproducibility Initiative (https://sc26.supercomputing.org/program/papers/reproducibility-initiative/) we require detailed and structured artifact descriptions (AD) using the SC26 format (https://github.com/jennfshr/sc26-repro) The AD should include a field for one or more links to data (Zenodo, figshare, etc.) and code (Github, GitLab, Bitbucket, etc.) repositories. For the artifacts that will be placed in the code repository, we encourage authors to follow the PDSW 2026 Reproducibility Addendum (https://www.pdsw.org/pdsw26/PDSW2026ReproducibilityInitiativeAddendum.pdf) on how to structure the artifact, as it will make it easier for the reviewing committee and readers of the paper in the future. Submissions website: https://submissions.supercomputing.org/ ----------------------------- Organization Team ----------------------------- General Chair: Jay Lofstead Sandia National Laboratories, USA Program Co-Chairs: Sarah Neuwirth Johannes Gutenberg University Mainz, Germany Lipeng Wan Georgia State University, USA Reproducibility Chair: Ricardo Macedo INESC TEC & University of Minho, Portugal Publicity Chair Radita Liem Johannes Gutenberg University Mainz, Germany Web & Publications Chair: Joan Digney Carnegie Mellon University, USA -------------------------------------------------------------------------------- Univ.-Prof. Dr. Sarah M. Neuwirth Co-Director, NHR South-West HPC Center Research Group Head, High Performance Computing and its Applications Johannes Gutenberg University Mainz Anselm-Franz-von-Bentzelweg 12 55099 Mainz | Germany Office: ZDV, Room 01-339 Phone: +49 6131 39 23643 Email: neuwirth(a)uni-mainz.de<mailto:neuwirth@uni-mainz.de> Website: https://www.hpca-group.de/ NHR@SW Website: https://nhrsw.de/ --------------------------------------------------------------------------------

1 0

June Ceph Developer Meetup
by Anthony Middleton 02 Jun '26

02 Jun '26

Hey Ceph Friends, I hope your week is off to a good start. Just a reminder that the next Ceph Developer Monthly Meetup is this Wednesday, June 3rd, at 8 am PDT. RSVP here: https://www.meetup.com/ceph-user-group/events/315060630/ Be sure to add your topic to the meeting tracker: https://tracker.ceph.com/projects/ceph/wiki/CDM_03_JUN_2026 Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 02 Jun '26

02 Jun '26

Hi, I must have followed the wrong instructions (as often seems to be the case when it comes to Ceph). I gave Ceph another try with --fsid, and then it worked. A missing fsid file cost me an hour, but the OSD subfolder seems to be OK now: mixtile@blade3n1:~$ sudo ls -al /var/lib/ceph/8aad3073-39a1-11f1-bf6e-f2704a1efa 9b/osd.2 total 16 drwxr-xr-x 2 ceph ceph 4096 Jun 1 16:50 . drwx------ 14 167 167 4096 Jun 1 13:47 .. lrwxrwxrwx 1 ceph ceph 93 Jun 1 16:44 block -> /dev/ceph-81bc1761-e8f6-446c- 96f3-eb1b8f92628b/osd-block-9f7fd40d-0698-40b9-8718-62942b03e263 -rw-r--r-- 1 ceph ceph 37 Jun 1 16:50 fsid In the meantime, I've managed to issue this command: sudo ceph-volume lvm activate --all There's one problem though: I get the feedback that ceph-osd@2 is working, but sudo systemctl status ceph-osd@2 says that the daemon failed to start: Jun 01 18:57:43 blade3n1 ceph-osd[317772]: 2026-06-01T18:57:43.403+0200 ffffb584 9040 -1 osd.2 7314 init authentication failed: (13) Permission denied What puzzles me: Why is the OSD daemon started as a normal service and not as a Docker container? Am So., 31. Mai 2026 um 21:27 Uhr schrieb Eugen Block via ceph-users < ceph-users(a)ceph.io>: > I don't see the '--fsid' flag in your command, it needs to be part of > the bootstrap command. So 'cephadm bootrap --mon-ip IP --fsid FSID' > and so on. > > 4 OSDs istn't that much, it should be doable in a reasonable amount of > time. It took me around two hours to bring my test cluster with 3 OSDs > back up, I think. > > > /var/lib/ceph/osd/ceph-2 > > This is the view from within the cephadm shell, the directory on the > filesystem is : > > /var/lib/ceph/{FSID}/osd.2 > > which is then mapped into the container. > > Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > > >> > >> No, the data is not inaccessible until you wipe the OSDs. > > > > > > Ah, that's good to know that at least the data are still there. > > > > How many OSDs are we talking about here? > > > > > > I've got 4 OSDs, which hold 1.3 TB of data in total. I do have a backup > of > > them on an external HDD, which spit out a handful of warnings during the > > last restore, so destroying the previous SSD, creating new ones, and > trying > > another restore are things I'd like to avoid. > > > > And regarding the fsid, you can specify it in the bootstrap command, > >> you won’t be able to change it afterwards. > > > > > > This is exactly what I tried to do: > > > > sudo cephadm bootstrap --mon-ip $MYIP --config /etc/ceph/ceph.conf~~ > > --allow-overwrite > > > > And /etc/ceph/ceph.conf~~ contains the old FSID: > > > > # minimal ceph.conf for 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > > [global] > > fsid = 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > > mon_host = [v2:10.20.0.11:3300/0,v1:10.20.0.11:6789/0] [v2: > > 10.20.0.12:3300/0,v1:10.20.0.12:6789/0] [v2: > > 10.20.0.13:3300/0,v1:10.20.0.13:6789/0] [v2: > > 10.20.0.14:3300/0,v1:10.20.0.14:6789/0] > > > > Nevertheless, cephadm ignored the old settings and gave the new cluster > a > > generated ID. So I've gotta tear down the cluster and start again? What > > about the key pairs stored in /etc/ceph? > > > > You can simply create the OSD directories and create a symbolic link to > >> the block device. Cephadm simply maps it then. > > > > > > I've now recreated the dir /var/lib/ceph/osd/ceph-2 and have to symlink > > /dev/ceph-81bc1761-e8f6-446c-96f3-eb1b8f92628b/o > > sd-block-9f7fd40d-0698-40b9-8718-62942b03e263 there? Or should ceph-2 be > > the name of the symlink? > > > > I recommend to hire a professional to help you out, just to be on the > safe > >> side. > > > > > > I've got a Ceph expert in our hackspace, but I'm not sure whether he can > > help me. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: tentacle 20.2.2 QE validation status
by Patrick Donnelly 01 Jun '26

01 Jun '26

I've created a tracker for this bug: https://tracker.ceph.com/issues/77037 On Mon, Jun 1, 2026 at 3:57 AM Guillaume ABRIOUX via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Yuri, > > few tests in the orch/cephadm suite may be failing due to issues in their definitions: nvme-loop setup appears to be affected by nvme-cli behavior, see [1] and [2] > > [1] https://tracker.ceph.com/issues/76184 > [2] https://github.com/ceph/ceph/pull/68569/changes#r3332344141 > > > Thanks! > > -- > Guillaume Abrioux > Software Engineer > From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> > Date: Tuesday, 26 May 2026 at 18:20 > To: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> > Subject: [EXTERNAL] [ceph-users] tentacle 20.2.2 QE validation status > > Details of this release are summarized here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya > nvmeof: Aviv, Vallari > > upgrade: Patrick > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > Unless otherwise stated above: > > Compagnie IBM France > Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex > RCS Nanterre 552 118 465 > Forme Sociale : S.A.S. > Capital Social : 664 614 175,50 € > SIRET : 552 118 465 03644 - Code NAF 6203Z > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0