- ceph-users - lists.ceph.io

[erasure-code] limits on mirroring
by garcetto 20 May '25

20 May '25

good morning, i would like to know if there are some limits/best practices or still not production ready support for ceph mirroring/journailng between two different pools types, say replica --> erasure-code pool or vice-versa. thank you.

1 0

Questions about managing RGW multisite
by Adam Prycki 19 May '25

19 May '25

Hello, I have few question about managing rgw multisite deployments. A) zonegroups and zone endpoints can have multiple URLs/addresses. Is it better to pass just one url to load balancer or configure separate address for each RGW daemon in the zone. B) Rook documentation suggest that having more than 1 RGW running replication is bad for performance. I've also seen some IBM blog posts discussing rgw multisite with graphics showing only single RGW running replication traffic. I don't think there are any warnings about this in official documentation. Is this something you consider when planning your multisite deployments? How many RGW dedicated to replication do you usually deploy per zone? https://www.rook.io/docs/rook/v1.17/Storage-Configuration/Object-Storage-RG… C) Afaik most radosgw-admin command should be executed on ceph cluster with master zonegroup. When I commit changes after these operation period doesn't seem to update on secondary zones. Is this supposed to work like this? D) How are we supposed to keep period on secondary zone in sync? If I need to change secondary zone endpoint I would have always pull the current realm from master zone? 1. pull the latest realm/period from master zone. 2. modifying the secondary zone zone 3. run radosgw-admin period update --commit from secondary zone. E) Do radosgw-admin commands modifying zone on secondary cluster require committing the period afterwards? (for example zone storage class pool change doesn't directly affect zonegroup ) Do they always require rgw restart? Example with enabling compression doesn't commit the period and doesn't mention restarting rgw. https://docs.ceph.com/en/latest/radosgw/compression/ Placement/Storage Classes example mentions that you have to commit period and restart rgw after change. https://docs.ceph.com/en/latest/radosgw/placement/#adding-a-storage-class F) Is it safe to restart RGW daemons during full zone sync? Can it interrupt or negatively affect zone synchronization? Best regards Adam Prycki

1 0

[cephadm] Questions regarding cephadm infra-as-code philosophy, containerization, and trixie availability
by Florent Carli 19 May '25

19 May '25

Hello ceph team, I’ve been working with ceph for some time and am currently evaluating cephadm as a successor to ceph-ansible. During this process, I’ve encountered a few aspects of the cephadm approach that raised some questions, and I’d greatly appreciate your insights to better understand the intended usage and philosophy. 1) Declarative workflows and Infrastructure as Code One of the advantages of ceph-ansible was the ability to define the cluster state declaratively in YAML files, which aligned well with Infrastructure-as-Code principles. With cephadm, the process appears more imperative and CLI-driven, which makes automation and reproducibility harder in comparison. Is there a recommended approach to achieving a fully declarative deployment model with cephadm? Or plans to support this more directly? 2) Containerization vs. local dependencies Cephadm’s move to full containerization makes sense in principle, especially to avoid system-level dependencies. However, in practice, many operations (e.g., using ceph-bluestore-tool, or the python modules for Rados/rbd) still seem to require installing packages on the host. Is this the expected model : containers for core daemons, but local packages for tooling ? It feels somewhat contradictory, and I wonder if there's a clearer pattern or guidance for those cases. 3) Ceph packages for Debian Trixie on download.ceph.com Since I'm using debian, I'm also in the process of anticipating the soon to come Debian 13 version (Trixie). "cephadm install" pulls packages from download.ceph.com, where the latest versions are published. Currently, it seems that there are no ceph packages yet for Trixie on download.ceph.com, which forces me to use the debian hosted packages (they are not even the same ceph version). Is there an ETA for when those packages will be made available? Or maybe there's another url for development packages? Thank you very much for your time and for your work on cephadm. I’m looking to better align our automation practices with the direction Cephadm is taking, and any clarification you can provide would be very helpful. Best regards, Florent.

8 8

Re: 19.2.1: filestore created when bluestore requested
by Konold, Martin 17 May '25

17 May '25

Am 2025-05-16 18:01, schrieb Anthony D'Atri: > I wouldn’t think blkdiscard would necessarily fully clean. I would try > sgdisk —zap-all or Ceph-volume lvm zap I gave this a try in addition to a reboot but no changes. Still not created a bluestore osd as intended. I guess this is the culprit: 2025-05-16T16:28:14.865+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] Maybe someone has a hint? Yours --martin ------------------------------------------------------------------------------ # ceph-volume lvm prepare --bluestore --data vgsdd/sdd1 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a5ae5940-26f2-44c8-bf16-bfdfa6de20ba Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdd/sdd1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-7 Running command: /usr/bin/ln -s /dev/vgsdd/sdd1 /var/lib/ceph/osd/ceph-1/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap stderr: got monmap epoch 20 --> Creating keyring file for osd.1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid a5ae5940-26f2-44c8-bf16-bfdfa6de20ba --setuser ceph --setgroup ceph stderr: 2025-05-16T16:28:14.865+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.865+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.866+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.867+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T16:28:14.867+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1//block) No valid bdev label found stderr: 2025-05-16T16:28:15.127+0000 7be7486fb880 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: vgsdd/sdd1 root@pve-03 (pve-03.t3):~# ceph-volume lvm activate --all Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key --> OSD ID 0 FSID a764e57e-ad3b-4823-8849-454f133526f6 process is active. Skipping activation --> Activating OSD ID 1 FSID a5ae5940-26f2-44c8-bf16-bfdfa6de20ba Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdd/sdd1 --path /var/lib/ceph/osd/ceph-1 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdd/sdd1 /var/lib/ceph/osd/ceph-1/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-7 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/systemctl enable ceph-volume@lvm-1-a5ae5940-26f2-44c8-bf16-bfdfa6de20ba stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-1-a5ae5940-26f2-44c8-bf16-bfdfa6de20ba.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@1 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)1.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@1 --> ceph-volume lvm activate successful for osd ID: 1 ceph-volume-systemd.log has: [2025-05-16 16:23:43,600][ceph_volume.process][INFO ] stderr --> RuntimeError: could not find osd.1 with osd_fsid a2a1cb42-006f-4659-a461-bb0ffbf190ff ps: ceph 11572 0.5 0.0 689688 83000 ? Ssl 16:29 0:02 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph # systemctl status ceph-osd(a)1.service ● ceph-osd(a)1.service - Ceph object storage daemon osd.1 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: enabled) Drop-In: /usr/lib/systemd/system/ceph-osd@.service.d └─ceph-after-pve-cluster.conf Active: active (running) since Fri 2025-05-16 16:29:11 UTC; 7min ago Process: 11567 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 1 (code=exited, status=0/SUCCESS) Main PID: 11572 (ceph-osd) Tasks: 68 Memory: 49.4M CPU: 2.239s CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)1.service └─11572 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph Mai 16 16:29:11 pve-03 systemd[1]: Starting ceph-osd(a)1.service - Ceph object storage daemon osd.1... Mai 16 16:29:11 pve-03 systemd[1]: Started ceph-osd(a)1.service - Ceph object storage daemon osd.1. Mai 16 16:29:15 pve-03 ceph-osd[11572]: 2025-05-16T16:29:15.066+0000 7b3558e37880 -1 osd.1 0 log_to_monitors true Mai 16 16:29:16 pve-03 ceph-osd[11572]: 2025-05-16T16:29:16.351+0000 7b3554dd76c0 -1 osd.1 0 waiting for initial osdmap Mai 16 16:29:16 pve-03 ceph-osd[11572]: 2025-05-16T16:29:16.352+0000 7b354c6166c0 -1 osd.1 0 failed to load OSD map for epoch 32827, got 0 bytes Mai 16 16:29:16 pve-03 ceph-osd[11572]: 2025-05-16T16:29:16.657+0000 7b3548b646c0 -1 osd.1 33967 osdmap NOUP flag is set, waiting for it to clear Mai 16 16:29:17 pve-03 ceph-osd[11572]: 2025-05-16T16:29:17.354+0000 7b3548b646c0 -1 osd.1 33968 osdmap NOUP flag is set, waiting for it to clear Mai 16 16:29:18 pve-03 ceph-osd[11572]: 2025-05-16T16:29:18.362+0000 7b3548b646c0 -1 osd.1 33969 osdmap NOUP flag is set, waiting for it to clear -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

3 2

19.2.1: filestore created when bluestore requested
by Konold, Martin 17 May '25

17 May '25

Hi, I am working on a small 3 node ceph cluster which used to work as expected. When creating a new ceph osd the ceph-volume command throws some errors and filestore instead of bluestore is created. (The drive was clean using blkdiscard before and no traces left in /etc from previous attempts. Any hints? Regards --martin # ceph-volume lvm create --bluestore --data vgsdb/sdb1 --> Incompatible flags were found, some values may get ignored --> Cannot use None (None) with --bluestore (bluestore) Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 19959f26-f2ed-451d-acb7-a808d15510cf Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -h ceph:ceph /dev/vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/ln -s /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory 2025-05-16T14:47:08.049+0000 772d282ef6c0 -1 AuthRegistry(0x772d200660d0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: got monmap epoch 20 --> Creating keyring file for osd.0 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 19959f26-f2ed-451d-acb7-a808d15510cf --setuser ceph --setgroup ceph stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-0//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] stderr: 2025-05-16T14:47:08.948+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found stderr: 2025-05-16T14:47:09.190+0000 7eaab06e0880 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: vgsdb/sdb1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vgsdb/sdb1 --path /var/lib/ceph/osd/ceph-0 --no-mon-config Running command: /usr/bin/ln -snf /dev/vgsdb/sdb1 /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-6 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-0-19959f26-f2ed-451d-acb7-a808d15510cf.service → /lib/systemd/system/ceph-volume@.service. Running command: /usr/bin/systemctl enable --runtime ceph-osd@0 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)0.service → /lib/systemd/system/ceph-osd@.service. Running command: /usr/bin/systemctl start ceph-osd@0 --> ceph-volume lvm activate successful for osd ID: 0 --> ceph-volume lvm create successful for: vgsdb/sdb1 -- Martin Konold - Prokurist, CTO KONSEC GmbH -⁠ make things real Amtsgericht Stuttgart, HRB 23690 Geschäftsführer: Andreas Mack Im Köller 3, 70794 Filderstadt, Germany

3 3

[reef] radosgw user has objects but no bucket
by Boris 16 May '25

16 May '25

Hi, I am in the process of cleaning things up and just saw that we have a user, that seems to have data but no buckets. # radosgw-admin user stats --uid 3853f960-6d72-4760-95ea-5e8f2a571840 --sync-stats { "stats": { "size": 189946686516, "size_actual": 189946687488, "size_kb": 185494812, "size_kb_actual": 185494812, "num_objects": 12069 }, "last_stats_sync": "2025-05-16T08:03:49.458563Z", "last_stats_update": "2018-02-09T14:17:26.334536Z" } # radosgw-admin bucket list --uid 3853f960-6d72-4760-95ea-5e8f2a571840 [] How can I find the users data, to verify if this can be removed? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

quincy 17.2.9 hotfix QE validation status
by Yuri Weinstein 15 May '25

15 May '25

Details of this release are summarized here: https://tracker.ceph.com/issues/71311#note-1 Release Notes - https://github.com/ceph/ceph/pull/63267 LRC upgrade - N/A Seeking approvals/reviews for: rados - Laura, Adam Kupczyk approved? This hotfix testing was limited to smoke and rados suites.

2 1

Scrubbing + Deep crubbing
by Devender Singh 15 May '25

15 May '25

Hello All Need some help to tune scrubbing. 1. How to control not run scrubbing and deep scrubbing together and which one should be started first? 124 active+clean+remapped 35 active+clean+scrubbing+deep 24 active+clean+scrubbing 4 active+clean+remapped+scrubbing+deep I have osd_max_scrubs =1 but still impacting IO’s special when both are running together. 2. When changing some priorities to not to impact IO’s, Setting higher number is lower priority or it works differently with different kind of priority parameters? What is the standard of defining high and low priority(higher number is lower priority or lower number is lower priory etc...) on CEPH? eg: osd_requested_scrub_priority - 170 (120 default) Eg: ceph config set osd osd_scrub_event_cost 8192 (4096 default) Regards Dev

2 2

OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors
by Alex 15 May '25

15 May '25

Hi everyone. Help me settle a debate. My coworker is seeing OBJECT_UNFOUND and PG_DAMAGED (recovery_unfound) errors. We both agree they are caused by bad drives. The fix is to mark the drive as out, replace it and add it back in. Whenever we see this error on Ceph we see corresponding read errors on the physical drive. I'm saying that even though the drive is bad since there are two more copies, only 1 of 3 drives has bad sectors preventing the data from being accessed which is what dmesg is showing ie critical medium error, dev sd..., sector 12345... There should not be OBJECT_UNFOUND since Ceph compares the remaining two copies and assuming the data matches, it should be able to recover on it's own and move the data to another PG or maybe OBJECT_UNFOUND and PG_DAMAGED are warnings not errors. My coworker is saying because the primary OSD responsible for coordinating the PG was the one which failed, and is the "source of truth" the cluster goes into error state. His argument doesn't make sense to me since there should be no single point of failure, but I'm also not sure about my argument since I don't know enough about how Ceph works under the hood. Thanks.

3 9

Repo name bug?
by Alex 15 May '25

15 May '25

Good morning everyone. Does the preflight playbook have a bug? https://github.com/ceph/cephadm-ansible/blob/devel/cephadm-preflight.yml Line 82: paths: "{{ ['noarch', '$basearch'] if ceph_origin == 'community' else ['$basearch'] }}" The yum repo file then gets named ceph_stable_$basearch. Shouldn't it be basearch without the $ ? - Alex

3 10