March 2026 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

cephadm without containers
by Adrian Sevcenco 10 Apr '26

10 Apr '26

Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian

11 20

Re: tentacle 20.2.1 QE validation status
by Adam Emerson 31 Mar '26

31 Mar '26

On 22/03/2026, Yuri Weinstein via dev wrote: > We have merged several important PRs and reran all suites. > > Details of this release are summarized here: > https://tracker.ceph.com/issues/74838#note-4 > > Seeking reviews and approval for all suites, please see runs on build 2 RGW Approved.

2 1

Rescheduled Ceph Developer Monthly EMEA Meetup
by Anthony Middleton 31 Mar '26

31 Mar '26

Hey Ceph Friends, This week's Ceph Developer Monthly EMEA Meetup has been moved to next week, April 8th. Please add topics and/or questions to the agenda for our next session - https://tracker.ceph.com/projects/ceph/wiki/CDM_01-APR-2026 I'm looking forward to a great discussion. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: tentacle 20.2.1 QE validation status
by Sridhar Seshasayee 31 Mar '26

31 Mar '26

Hi Yuri, Rados: Approved upgrade/squid-x (tenacle) - Approved upgrade/reef-x (tenacle) - One failure is under investigation. See: https://tracker.ceph.com/issues/75570 upgrade/telemetry-upgrade - Approved Please see https://tracker.ceph.com/projects/rados/wiki/TENTACLE#v2021-httpstrackercep… for a detailed summary. -Sridhar

1 0

Re: tentacle 20.2.1 QE validation status
by Yuri Weinstein 30 Mar '26

30 Mar '26

Action required Missing approvals for: rados - Yaarit, Laura rgw - Adam E orch - Redouane ceph-volume - Guillaume upgrade/squid-x (tenacle) - Laura upgrade/reef-x (tenacle) - Laura upgrade/telemetry-upgrade - Laura On Sun, Mar 22, 2026 at 8:17 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > We have merged several important PRs and reran all suites. > > Details of this release are summarized here: > https://tracker.ceph.com/issues/74838#note-4 > > Seeking reviews and approval for all suites, please see runs on build 2 > > On Mon, Mar 9, 2026 at 11:44 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/74838#note-4 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya (still running) > > nvmeof: Aviv, Vallari > > > > upgrade/squid-x: Laura > > upgrade/reef-x: Laura > > upgrade/telemetry-upgrade: Laura > > > > upgrade/tentacle-p2p: N/A > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 1

Re: tentacle 20.2.1 QE validation status
by Venky Shankar 30 Mar '26

30 Mar '26

Hi Yuri, On Tue, Mar 17, 2026 at 2:22 AM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > Only dashboard, rbd and krbd were approved. > > Patrick is working on https://github.com/ceph/ceph/pull/67750 revert > > I also added tentacle-p2p run and it needs reviews. > > Dev leads, pls review/approve the run results. > > On Mon, Mar 9, 2026 at 11:44 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/74838#note-4 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky The fs suite runs looks good. The initial run https://pulpito.ceph.com/yuriw-2026-03-04_19:10:49-fs-tentacle-release-dist… Had an infrastructure issue, and many jobs failed as per below 500 Server Error: Internal Server Error for url: https://2.chacra.ceph.com/repos/ceph/tentacle/70f8415b300f041766fa27faf7d54… I reran the failed jobs with tentacle-release branch here https://pulpito.ceph.com/vshankar-2026-03-26_07:36:37-fs-tentacle-release-d… I have filed a tracker for one new failure under the Linux Kernel Client project https://tracker.ceph.com/issues/75735 The test failure above does not block the run verification. Therefore, I am approving the run: fs approved - failures are https://tracker.ceph.com/projects/cephfs/wiki/Tentacle#v2021 > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya (still running) > > nvmeof: Aviv, Vallari > > > > upgrade/squid-x: Laura > > upgrade/reef-x: Laura > > upgrade/telemetry-upgrade: Laura > > > > upgrade/tentacle-p2p: N/A > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 0

Re: tentacle 20.2.1 QE validation status
by Guillaume ABRIOUX 30 Mar '26

30 Mar '26

Hi Yuri, ceph-volume approved Thanks! -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Monday, 30 March 2026 at 16:08 To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] Re: tentacle 20.2.1 QE validation status Action required Missing approvals for: rados - Yaarit, Laura rgw - Adam E orch - Redouane ceph-volume - Guillaume upgrade/squid-x (tenacle) - Laura upgrade/reef-x (tenacle) - Laura upgrade/telemetry-upgrade - Laura On Sun, Mar 22, 2026 at 8:17 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > We have merged several important PRs and reran all suites. > > Details of this release are summarized here: > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Seeking reviews and approval for all suites, please see runs on build 2 > > On Mon, Mar 9, 2026 at 11:44 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > Details of this release are summarized here: > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya (still running) > > nvmeof: Aviv, Vallari > > > > upgrade/squid-x: Laura > > upgrade/reef-x: Laura > > upgrade/telemetry-upgrade: Laura > > > > upgrade/tentacle-p2p: N/A > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

ACME for the mgmt-gateway
by Matthias Riße 30 Mar '26

30 Mar '26

Hey all! I've just managed to get the mgmt-gateway working on our Ceph Cluster with cephadm, which is quite cool to have as a single HA entrypoint. I am surprised about the lack of documentation to get ACME-signed certificates into / in front of the mgmt-gateway. Is anyone doing this already? If yes, what's your setup? I am wondering if it is easier to put certbot, lego, or similar into a "custom container service" and update the mgmt-gateway service with renewed certs from there, or run an additional caddy instance (also as a custom container service?) as a reverse proxy in front of the mgmt-gateway and let it handle ACME. Kind regards Matthias -- --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellvertretende Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

1 0

Re: Restricting subuser to its bucket
by Eugen Block 30 Mar '26

30 Mar '26

Hi, I would suggest to run the aws cp command with --debug to see more details, the message "argument of type 'NoneType' is not iterable" is not really explanatory. ;-) Regards, Eugen "Rok Jaklič via ceph-users" ceph-users(a)ceph.io – 18. März 2026 10:53 > Hi, > > I am trying to restrict subuser to its bucket (so he/she can work only on a > bucket which has permission) with disabling ACLs and setting policy on a > bucket: > > Using ceph version 19.2.2 > > bucket1-policy.json contains > { > "Version": "2012-10-17", > "Statement": [ > { > "Effect": "Allow", > "Principal": { > "AWS": "arn:aws:iam::tenant1:user/parent1:user1" > }, > "Action": "s3:*", > "Resource": [ > "arn:aws:s3:::bucket1", > "arn:aws:s3:::bucket1/*" > ] > } > ] > } > > --- > > echo "lala" > test.txt > > export RGW_ENDPOINT="" > > radosgw-admin user create --tenant=tenant1 --uid=parent1 --key-type=s3 > --display-name="tenant1:parent1" > for sub in user1 user2 user3; do > radosgw-admin subuser create --tenant=tenant1 --uid=parent1 --subuser=$sub > --key-type=s3 --access=full; > done > > export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user1").access_key') > export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user1").secret_key') > > aws s3api create-bucket --bucket bucket1 --object-ownership > BucketOwnerEnforced --endpoint-url $RGW_ENDPOINT > aws s3api put-bucket-policy --bucket bucket1 --policy > file://bucket1-policy.json --endpoint-url $RGW_ENDPOINT > aws s3api put-public-access-block --bucket bucket1 > --public-access-block-configuration > BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true > --endpoint-url $RGW_ENDPOINT > > aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT > aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT > > # verify > export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user2").access_key') > export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user2").secret_key') > > aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT > > --- > > Script stops at aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url > $RGW_ENDPOINT with an error like: > upload failed: ./test.txt to s3://bucket1/test.txt argument of type > 'NoneType' is not iterable > > If I comment the line with aws s3api put-public-access-block... I can > successfully upload some file, but I can also see the contents of bucket1 > with user2, which is not what I want. > > Any ideas how would I restrict the subuser to its bucket only? > > I was actually able to restrict subuser to its bucket by explicitly denying > access to bucket with following policy: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/AFNERFMNSC… > > But it doesn't feel the right way. Wouldn't be better to "restrict" > everything for everyone but its user in the first place and then allow > others to access? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Crash on newly created NVMe+HDD OSD
by Jan Kasprzak 30 Mar '26

30 Mar '26

Hello, Ceph users, I wanted to replace a HDD with bad blocks in an OSD which has block.db on NVMe. Removing the OSD (ceph-volume lvm zap, etc.) went well, no VG/LV remained for data or metadata. But after replacing the HDD, preparing a new HDD+NVMe OSD, and starting it with "ceph-volume lvm activate --all", the OSD daemon crashed with terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input Retrying the proces once more I noticed that ceph-volume created a VG/PV/LV on the HDD (/dev/sda), but did not create a VG/PV/LV on /dev/nvme1n1p4. Zapping the result once more and creating a HDD-only OSD helped, the OSD daemon started and joined the cluster correctly. I then stopped it once more, and tried to create a db volume myself, and migrate db to it: systemctl stop ceph-osd@$OSD_ID NEW_DB_VG=ceph-`uuidgen` NEW_DB_LV=osd-db-`uuidgen` vgcreate $NEW_DB_VG /dev/nvme1n1p4 lvcreate -a y -l 100%FREE -n $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm new-db --osd-id $OSD_ID --osd-fsid $OSD_FSID --target $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm migrate --osd-id $OSD_ID --osd-fsid $OSD_FSID --from data --target $NEW_DB_VG/$NEW_DB_LV systemctl start ceph-osd@$OSD_ID ... and it started correctly and did not crash. I used the same commands previously on Ceph 19 when replacing HDDs, and it worked without problem. This is the first time I replaced a HDD after upgrading to Ceph 20.2.0, so it might be a regression in ceph-volume of 20.2.0. Below is the output of "ceph-volume lvm prepare", and the crash report from journal. Did I do something incorrect here? Thanks! -Yenya # ceph-volume lvm prepare --bluestore --block.db /dev/nvme1n1p4 --data /dev/sda Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: vgcreate --force --yes ceph-8805a195-57fd-4054-8282-a307d15955a5 /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-8805a195-57fd-4054-8282-a307d15955a5" successfully created Running command: lvcreate --yes -l 2861055 -n osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 ceph-8805a195-57fd-4054-8282-a307d15955a5 stdout: Logical volume "osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963" created. Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-25 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 Running command: /usr/bin/ln -s /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 /var/lib/ceph/osd/ceph-25/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-25/activate.monmap stderr: got monmap epoch 39 --> Creating keyring file for osd.25 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/keyring Running command: /usr/bin/chown -R ceph:ceph /dev/nvme1n1p4 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 25 --monmap /var/lib/ceph/osd/ceph-25/activate.monmap --keyfile - --bluestore-block-db-path /dev/nvme1n1p4 --osd-data /var/lib/ceph/osd/ceph-25/ --osd-uuid db38dbfb-e161-4680-93e3-ba4abaa36963 --setuser ceph --setgroup ceph stderr: 2026-03-30T07:59:06.150+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25//block) No valid bdev label found stderr: 2026-03-30T07:59:06.175+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/sda Mar 30 07:52:31 osdhost systemd[1]: Starting Ceph object storage daemon osd.25... Mar 30 07:52:31 osdhost systemd[1]: Started Ceph object storage daemon osd.25. Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' Mar 30 07:52:31 osdhost ceph-osd[1140285]: what(): static void bluefs_fnode_t::_denc_finish(ceph::buffer::v15_2_0::ptr::const_iterator&, char**, uint32_t*): Malformed input [buffer:3] Mar 30 07:52:31 osdhost ceph-osd[1140285]: *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Main process exited, code=dumped, status=6/ABRT Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Failed with result 'core-dump'. -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Update of debian packages from 18.2.7
by Bandelow, Gunnar 27 Mar '26

27 Mar '26

Hello, i try to update the ceph packages on debian 12 (bookworm). apt list --upgradable Listing... Done ceph-common/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] cephadm/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] docker-ce-cli/bookworm 5:29.3.1-1~debian.12~bookworm amd64 [upgradable from: 5:29.3.0-1~debian.12~bookworm] docker-ce/bookworm 5:29.3.1-1~debian.12~bookworm amd64 [upgradable from: 5:29.3.0-1~debian.12~bookworm] docker-compose-plugin/bookworm 5.1.1-1~debian.12~bookworm amd64 [upgradable from: 5.1.0-1~debian.12~bookworm] libcephfs2/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] librados2/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] libradosstriper1/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] librbd1/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] librgw2/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-ceph-argparse/stable 18.2.8-1~bpo12+1 all [upgradable from: 18.2.7-1~bpo12+1] python3-ceph-common/stable 18.2.8-1~bpo12+1 all [upgradable from: 18.2.7-1~bpo12+1] python3-cephfs/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-rados/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-rbd/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-rgw/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] These packages are available for an update. Unfortunately, the ceph-common package seems to depend on the wrong packages version: apt full-upgrade Reading package lists... Done Building dependency tree... Done Reading state information... Done Calculating upgrade... Error! Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: ceph-common : Depends: librbd1 (= 18.2.7-1~bpo12+1) but it is not going to be installed Depends: python3-rbd (= 18.2.7-1~bpo12+1) but it is not going to be installed E: Unable to correct problems, you have held broken packages. Any suggestions? Best regards Gunnar ======================================================= Universitätsrechenzentrum (URZ) Universität Greifswald Felix-Hausdorff-Straße 18 17489 GreifswaldGermany 📞: +49 3834 420 1450 🗨: @mxid-69420:matrix.uni-greifswald.de

1 0

Re: Is it possible to speedup cephadm OSD upgrades?
by Redouane Kachach 26 Mar '26

26 Mar '26

Hi Adam, Short answer is no at this moment. But there's already some good WIP related to that: https://github.com/ceph/ceph/pull/66108 https://github.com/ceph/ceph/pull/67977 Once this new feature is implemented (likely as part of the next Umbrella release) the upgrade process should go faster. Best, Redouane. On Thu, Mar 26, 2026 at 2:26 PM Adam Prycki via ceph-users < ceph-users(a)ceph.io> wrote: > Hello, > > is there a way to speedup OSD redeploy speed during cephadm cluster > upgrade? > > We are currently upgrading our large cluster and going really slowly. > "osd": { > "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) > squid (stable)": 2228, > "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > squid (stable)": 119 > }, > > At current pace upgrade may take a week because cephadm redeploys OSDs > one at a time. > > Is there some way to speed it up? Maybe redeploy OSDs one server at a time? > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Is it possible to speedup cephadm OSD upgrades?
by Adam Prycki 26 Mar '26

26 Mar '26

Hello, is there a way to speedup OSD redeploy speed during cephadm cluster upgrade? We are currently upgrading our large cluster and going really slowly. "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2228, "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 119 }, At current pace upgrade may take a week because cephadm redeploys OSDs one at a time. Is there some way to speed it up? Maybe redeploy OSDs one server at a time? Best regards Adam Prycki

1 0

Agents constantly contacting wrong mgr
by Francesco Di Nucci 26 Mar '26

26 Mar '26

Hi, I'm using Ceph Squid installed with cephadm and I'm having an issue with agents. Cluster health constantly goes in warn as agent do not report to the active mgr - on the other side agents report "Failed to send metadata to mgr: non-200 response <-1> from agent endpoint: [Errno 111] Connection refused". Inspecting agent activity with strace, they're all trying to contact monmgr01.example.com/X.Y.Z.1 , while actually the active manager is monmgr02.example.com/X.Y.Z.1. The issue remains if I redeploy the agents with ceph orch. Any suggestion on how to find out more on the issue? Has it happened to someone else? Thanks -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

1 0

Re: Issues upgrading cephadm mixed intel/ARM64 cluster
by Adam Prycki 25 Mar '26

25 Mar '26

Thank you, I didn't know about this config option. With it disabled now ceph orch upgrade status shows: "target_image": "quay.io/ceph/ceph:v19.2.3", MGR and MON have updated successfully. Cephadm uses the right string for container version. Hopefully we won't have any more issues with this upgrade. Best regards Adam Prycki On 25/03/2026 15:09, Eugen Block via ceph-users wrote: > Hi, > > you could try to disable use_repo_digest: > > ceph config help mgr/cephadm/use_repo_digest > mgr/cephadm/use_repo_digest - Automatically convert image tags to image > digest. Make sure all daemons use the same image > (bool, advanced) > Default: true > Can update at runtime: false > > You would need to restart the MGRs to apply that config update. > > Regards, > Eugen > > Zitat von Adam Prycki via ceph-users <ceph-users(a)ceph.io>: > >> Hello, >> >> we are having issues upgrading from 19.2.2 to 19.2.3 >> >> We have mixed ARM64 AMD64 ceph deployment. >> For performance reason we had to host our MON, MGR and MDS daemons on >> intel server. >> >> It didn't cause any issues for the past 3 years but today we >> discovered that we are not able to perform an upgrade. >> >> Before starting the upgrade, I manually pulled ceph image on all of >> our servers with podman pull. (podman pull quay.io/ceph/ceph:v19.2.3) >> Every server has the new image. >> >> We start upgrade with normal cephadm command: >> ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> >> Cephadm always fails on the first host with "UPGRADE_FAILED_PULL: >> Upgrade: failed to pull target image" >> >> ceph orch upgrade status says that ceph performs upgrade to >> "target_image": "quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717", >> not to specified quay.io/ceph/ceph:v19.2.3 >> >> 865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 >> appears to be ARM64 version of the container. https://quay.io/ >> repository/ceph/ceph/manifest/ >> sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee >> >> ARM64 container fails to start on AMD64 machines and upgrade can never >> start. >> >> 2026-03-25 12:27:15,337 7f98685251c0 DEBUG >> -------------------------------------------------------------------------------- >> cephadm ['--image', 'quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495', 'pull'] >> 2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image >> quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717... >> 2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout >> 23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75, >> [quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717] >> 2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from / >> usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host -- >> entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 --version >> 2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR >> (catatonit:2): failed to exec pid1: Exec format error >> 2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG >> -------------------------------------------------------------------------------- >> >> Is this a known issue? Has there been a change to how cephadm handles >> image version? >> >> We have made multiple upgrade of this cluster in the past and this is >> the first time we are seeing this issue. >> ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) >> >> Is there a way to force cephadm to use image tags instead if digests. >> >> Best regards >> Adam Prycki >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issues upgrading cephadm mixed intel/ARM64 cluster
by John Mulligan 25 Mar '26

25 Mar '26

On Wednesday, March 25, 2026 10:01:55 AM Eastern Daylight Time Adam Prycki via ceph-users wrote: > ---- > > Is this a known issue? Has there been a change to how cephadm handles > image version? This is a known issue. I can't find a tracker right now but this has been discussed multiple times in the past and an approach was designed to resolve the issue but AFAIK no one has implemented it. This is not new. I think you are just lucky not to have seen this before. > > We have made multiple upgrade of this cluster in the past and this is > the first time we are seeing this issue. > ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) > > Is there a way to force cephadm to use image tags instead if digests. > Eugen Block has shared the right answer in another reply to your message. Good luck.

1 0

Re: Issues upgrading cephadm mixed intel/ARM64 cluster
by Eugen Block 25 Mar '26

25 Mar '26

Hi, you could try to disable use_repo_digest: ceph config help mgr/cephadm/use_repo_digest mgr/cephadm/use_repo_digest - Automatically convert image tags to image digest. Make sure all daemons use the same image (bool, advanced) Default: true Can update at runtime: false You would need to restart the MGRs to apply that config update. Regards, Eugen Zitat von Adam Prycki via ceph-users <ceph-users(a)ceph.io>: > Hello, > > we are having issues upgrading from 19.2.2 to 19.2.3 > > We have mixed ARM64 AMD64 ceph deployment. > For performance reason we had to host our MON, MGR and MDS daemons > on intel server. > > It didn't cause any issues for the past 3 years but today we > discovered that we are not able to perform an upgrade. > > Before starting the upgrade, I manually pulled ceph image on all of > our servers with podman pull. (podman pull > quay.io/ceph/ceph:v19.2.3) Every server has the new image. > > We start upgrade with normal cephadm command: > ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 > > Cephadm always fails on the first host with "UPGRADE_FAILED_PULL: > Upgrade: failed to pull target image" > > ceph orch upgrade status says that ceph performs upgrade to > "target_image": > "quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717", > not to specified quay.io/ceph/ceph:v19.2.3 > > 865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 > appears to be ARM64 version of the container. > https://quay.io/repository/ceph/ceph/manifest/sha256:af0c5903e901e329adabe2… > > ARM64 container fails to start on AMD64 machines and upgrade can never start. > > 2026-03-25 12:27:15,337 7f98685251c0 DEBUG > -------------------------------------------------------------------------------- > cephadm ['--image', > 'quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495', > 'pull'] > 2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image > quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717... > 2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout > 23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75,[quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717] > 2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from > /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host > --entrypoint ceph --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 > --version > 2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR > (catatonit:2): failed to exec pid1: Exec format error > 2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG > -------------------------------------------------------------------------------- > > Is this a known issue? Has there been a change to how cephadm > handles image version? > > We have made multiple upgrade of this cluster in the past and this > is the first time we are seeing this issue. > ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) > > Is there a way to force cephadm to use image tags instead if digests. > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issues upgrading cephadm mixed intel/ARM64 cluster
by Adam Prycki 25 Mar '26

25 Mar '26

Hello, we are having issues upgrading from 19.2.2 to 19.2.3 We have mixed ARM64 AMD64 ceph deployment. For performance reason we had to host our MON, MGR and MDS daemons on intel server. It didn't cause any issues for the past 3 years but today we discovered that we are not able to perform an upgrade. Before starting the upgrade, I manually pulled ceph image on all of our servers with podman pull. (podman pull quay.io/ceph/ceph:v19.2.3) Every server has the new image. We start upgrade with normal cephadm command: ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 Cephadm always fails on the first host with "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" ceph orch upgrade status says that ceph performs upgrade to "target_image": "quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717", not to specified quay.io/ceph/ceph:v19.2.3 865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 appears to be ARM64 version of the container. https://quay.io/repository/ceph/ceph/manifest/sha256:af0c5903e901e329adabe2… ARM64 container fails to start on AMD64 machines and upgrade can never start. 2026-03-25 12:27:15,337 7f98685251c0 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495', 'pull'] 2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717... 2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout 23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75,[quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717] 2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 --version 2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR (catatonit:2): failed to exec pid1: Exec format error 2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG -------------------------------------------------------------------------------- Is this a known issue? Has there been a change to how cephadm handles image version? We have made multiple upgrade of this cluster in the past and this is the first time we are seeing this issue. ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) Is there a way to force cephadm to use image tags instead if digests. Best regards Adam Prycki

1 0

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

CSC minutes 2026-03-23
by Yaarit Hatuka 23 Mar '26

23 Mar '26

- (bumped from last week) [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe - why? this can be a solution to some technical decision making deadlocks we have from time to time - came from this discussion: https://ceph-storage.slack.com/archives/C05R975114L/p1772629034963979 - next step: feedback pls -> get the text in shape to vote on -> csc vote -> merge. - Should we consider a mailing list platform migration from postorius to groups.io - +1! - we will get support from LF for this as well - this also reduces the overhead to maitaining the mailing list imo - Ceph Foundation Board and CSC Q1 Meeting on March 30, 10 am PT - Ceph on Windows: Plan of action (Alexandre JARDON) - rocky10 PRs are merged to main - March Ceph Tech Talk speaker needed

1 0

tentacle 20.2.1 QE validation status
by Yuri Weinstein 22 Mar '26

22 Mar '26

Details of this release are summarized here: https://tracker.ceph.com/issues/74838#note-4 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya (still running) nvmeof: Aviv, Vallari upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: Laura upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 2

Re: Merging two clusters
by Nico Schottelius 21 Mar '26

21 Mar '26

Ciao Burkhard, Burkhard Linke via ceph-users <ceph-users(a)ceph.io> writes: > Hi, > > > we used the rbd live migration in the past to split up a cluster. The > processed pools were used for image and volumes in our openstack > cloud. Similar here, just using opennebula. > Live migration mostly works, but if the RBD image is in use, you have > to add some extra effort. Cinder for examples stores the mon > information in its database, and this has to be updated. Similar, plus with opennebula it's embedded in the libvirt XML and not so easy to modify. > I used a > script to migrate a RBD image, pause the instance, adjust the database > entries, and live migrate the instance itself to another host to > update the qemu/libvirt information. If the image is not actively used > migration is trivial; if the image is attached to a stopped virtual > machine migration is also trivial, but database entries have to be > updated. Got it. > The rather old nautilus clusters might be problem, so if you want to > try rbd live migration you need to test this first. Will take that into consideration. The rbd mirror might actually also work for us, because a short downtime can likely be tolerated. > Not sure whether the script still exists, but I can try to dig it out > if necessary. If you still have it, would certainly be interested to take a glimpse at it, at least for inspiration. Sunny greetings from Switzerland, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: Merging two clusters
by Nico Schottelius 21 Mar '26

21 Mar '26

Hey Anthony, Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> writes: > I’ve used rbd-mirror to do this sort of thing in the past. It’s > straightforward with unattached volumes. I had a homebrew harness > that for attached images would handle a reboot of the client with > reattach. That seems to be a way forward from what it seems. There are quite some details to address anyway from what I can see, as clients also need to be reconfigured and partially moved to a different network. Thanks for the pointer! Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Re: v18.2.8 Reef released
by Eugen Block 21 Mar '26

21 Mar '26

Hi, I just tried the first upgrade in a single node cluster (18.2.7 --> 18.2.8) and would like to share a few notes. The staggered upgrade should handle non-existing daemons a bit smoother. Before upgrading OSDs one has to upgrade the crash service first: ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.8 --daemon-types crash --limit 1 Initiating upgrade to quay.io/ceph/ceph:v18.2.8 And then the upgrade fails: ceph health detail HEALTH_ERR Upgrade: failed due to an unexpected exception [ERR] UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception Unexpected exception occurred during upgrade process: division by zero This is because there are no crash services running. Not sure how common this is, but maybe another check is required if there even are daemons of the specified type. Or at least add some details to the health output. Besides that, the limit parameter doesn't apply to all services. I reported that a few weeks ago (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/6TF5EOBGWC…) Apart from that, the upgrade was successful on my lab cluster, so thanks for the great work! Regards, Eugen Zitat von Yuri Weinstein via ceph-users <ceph-users(a)ceph.io>: > We're happy to announce the 8th, and expected to be last, backport release > in the Reef series. We recommend that all users update to this release. > > We recommend users to update to this release. > For detailed release notes with links & changelog please refer to the > official blog entry at > https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/ > > Known Issues > ------------ > * During QA for v18.2.8, it was found that there was a bug for upgrades from > Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a > deprecated connection feature bit that was adopted to indicate a Reef OSD. > This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually > upgraded to Reef. There are no known issues associated with Pacific and > Reef > OSDs interoperating where Pacific OSDs are "advertising" Reef > compatibility; > however, out of an abundance of caution, we no longer recommend upgrading > from Pacific to Reef directly. > > Security Fixes > -------------- > * CephFS Client: A fix was merged to prohibit unprivileged users from > modifying > the sgid or suid bits on a file. Previously, unprivileged users were > inadvertently permitted to set these bits if they were the sole bits being > modified. > * Mgr Alerts: The SMTP SSL context was enforced in the mgr/alerts module to > resolve a security vulnerability (GHSA-xj9f-7g59-m4jx). > > Notable Changes > --------------- > * RGW (RADOS Gateway): > - Fixed an issue where bucket rm --bypass-gc was mistakenly removing head > objects instead of tail objects, potentially causing data inconsistencies. > - Fixed rgw-restore-bucket-index to handle objects with leading hyphens > and to process versioned buckets correctly. > - Addressed an issue in the msg/async protocol that caused memory locks > and hangs during connection shutdown. > - RGW STS: Made JWKS URL verification configurable for AWS compliance via > the rgw_enable_jwks_url_verification configuration. > > * CephFS / MDS: > - Prevented the MDS from stalling (up to 5 seconds) during rename/stat > workloads by forcing the log to nudge for unstable locks after early > replies. > - Fixed cephfs-journal-tool so it no longer incorrectly resets the > journal trim position during disaster recovery, which was causing stale > journal objects to linger forever in the metadata pool. > - Fixed a bug where ll_walk incorrectly processed absolute paths as > relative paths. > - Prevented the ceph fs volume create command from accidentally deleting > user-created pools if the command aborted during cleanup. > - MDS Batched Operations: Added a new mds_allow_batched_ops configuration > option (default: true) to control whether the MDS can batch lookup or > getattr RPCs. > - CephFS Subvolumes: Added the ceph fs subvolume snapshot getpath command > to allow users to retrieve the absolute path of a snapshot of a subvolume. > > * BlueStore: > - Fixed a bug where the bytes_written_slow performance counter > incorrectly reported 0 when using aio_write. > > Getting Ceph > ------------ > * Git at git://github.com/ceph/ceph.git > * Tarball at https://download.ceph.com/tarballs/ceph-18.2.7.tar.gz > * Containers at https://quay.io/repository/ceph/ceph > * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ > * Release git sha1: efac5a54607c13fa50d4822e50242b86e6e446df > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: 8443: connect: connection refused
by Eugen Block 21 Mar '26

21 Mar '26

I haven't used mgmt-gateway yet, except for a short test if I could deploy it. But the fact that alertmanager is trying to connect to the wrong dashboard port makes me wonder. Alertmanager should be configured accoording to the dashboard config, for example I bootstrapped a new cluster with the parameter "--ssl-dashboard-port 8080": root@tentacle-ubuntu24:~# ceph mgr services { "dashboard": "https://192.168.124.172:8080/" "prometheus": "http://192.168.124.172:9283/" } After applying alertmanager, the webhook_configs are configured correctly: root@tentacle-ubuntu24:~# grep -A1 webhook_configs /var/lib/ceph/70495e92-2466-11f1-96c3-fa163edcc19e/alertmanager.tentacle-ubuntu24/etc/alertmanager/alertmanager.yml webhook_configs: - name: 'ceph-dashboard' webhook_configs: - url: 'https://tentacle-ubuntu24.domain:8080/api/prometheus_receiver' Did you see anything in the mgr logs related to "reconfigure" the alertmanager? Or did you maybe at some point change the ssl port of the dashboard (manually)? The respective config-key is: config/mgr/mgr/dashboard/ssl_server_port And does the error go away if you run: ceph orch reconfigure alertmanager Or alternatively redeploy it: ceph orch redeploy alertmanager Regards, Eugen Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > I did not think that ceph should be aware of any "external" proxy > being used - it runs on one of ceph nodes - I assumed such a proxy > would only be relevant to users/clients looking at ceph via > web-browser. > > As far a ceph goes I did: > > -> $ ceph config set mgr mgr/dashboard/standby_behaviour "redirect" > # set it back to default > > -> $ ceph config set mgr mgr/dashboard/standby_behaviour "error" > -> $ ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr > true # resolve to hostnames > -> $ ceph config set mgr mgr/dashboard/ssl false > -> $ ceph mgr services > { > "dashboard": "http://10.1.1.63:8080/", > "prometheus": "http://10.1.1.63:9283/" > } > > > Rest is Nginx's, what I'd call a "standard" proxy configuration. > If needed - I should spam it here? > > thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

v18.2.8 Reef released
by Yuri Weinstein 21 Mar '26

21 Mar '26

We're happy to announce the 8th, and expected to be last, backport release in the Reef series. We recommend that all users update to this release. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v18-2-8-reef-released/ Known Issues ------------ * During QA for v18.2.8, it was found that there was a bug for upgrades from Pacific to Reef. Pacific OSDs (and other Ceph daemons) were still using a deprecated connection feature bit that was adopted to indicate a Reef OSD. This can cause a OSD_UPGRADE_FINISHED warning before all OSDs are actually upgraded to Reef. There are no known issues associated with Pacific and Reef OSDs interoperating where Pacific OSDs are "advertising" Reef compatibility; however, out of an abundance of caution, we no longer recommend upgrading from Pacific to Reef directly. Security Fixes -------------- * CephFS Client: A fix was merged to prohibit unprivileged users from modifying the sgid or suid bits on a file. Previously, unprivileged users were inadvertently permitted to set these bits if they were the sole bits being modified. * Mgr Alerts: The SMTP SSL context was enforced in the mgr/alerts module to resolve a security vulnerability (GHSA-xj9f-7g59-m4jx). Notable Changes --------------- * RGW (RADOS Gateway): - Fixed an issue where bucket rm --bypass-gc was mistakenly removing head objects instead of tail objects, potentially causing data inconsistencies. - Fixed rgw-restore-bucket-index to handle objects with leading hyphens and to process versioned buckets correctly. - Addressed an issue in the msg/async protocol that caused memory locks and hangs during connection shutdown. - RGW STS: Made JWKS URL verification configurable for AWS compliance via the rgw_enable_jwks_url_verification configuration. * CephFS / MDS: - Prevented the MDS from stalling (up to 5 seconds) during rename/stat workloads by forcing the log to nudge for unstable locks after early replies. - Fixed cephfs-journal-tool so it no longer incorrectly resets the journal trim position during disaster recovery, which was causing stale journal objects to linger forever in the metadata pool. - Fixed a bug where ll_walk incorrectly processed absolute paths as relative paths. - Prevented the ceph fs volume create command from accidentally deleting user-created pools if the command aborted during cleanup. - MDS Batched Operations: Added a new mds_allow_batched_ops configuration option (default: true) to control whether the MDS can batch lookup or getattr RPCs. - CephFS Subvolumes: Added the ceph fs subvolume snapshot getpath command to allow users to retrieve the absolute path of a snapshot of a subvolume. * BlueStore: - Fixed a bug where the bytes_written_slow performance counter incorrectly reported 0 when using aio_write. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: efac5a54607c13fa50d4822e50242b86e6e446df

1 0

March Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 20 Mar '26

20 Mar '26

Hey folks, We will be having a Ceph science/research/big cluster call on Tuesday March 24th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: March 24th, 2026 9am Central US 14:00 UTC 3pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20260324 <https://pad.ceph.com/p/Ceph_Science_User_Group_20260324> Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Re: Recommended approach for hybrid devices
by Anthony D'Atri 20 Mar '26

20 Mar '26

Color me unamused. Reminds me of the Adaptec ACB 4000, which Sun's kernel detected by looking for an empty inquiry string. > On Mar 20, 2026, at 10:05 AM, Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > > I have encountered devices that claim to support multiple namespaces but do not implement the NVMe spec cleanly and send the same ID for each namespace. This causes the kernel to just show the first namespace.

1 0

Re: Recommended approach for hybrid devices
by Anthony D'Atri 20 Mar '26

20 Mar '26

> >> My understanding is a manual device add approach, while possible, removes >> the benefits of orch (creating "unmanaged" devices) and should be avoided. >> But would it even be possible for orch to be fed a yaml with the NVMes >> operating as both a data and db device? > > Check if the NVMe devices support multiple namespaces: Don't they all? > > https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html > > This way the kernel sees two block devices for each NVMe and the orchestrator is able to provision an OSD on one and the RocksDBs for the hybrid OSDs on the other. I've wondered for years how namespaces in pragmatic terms differ from partitions. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Recommended approach for hybrid devices
by Robert Sander 20 Mar '26

20 Mar '26

Hi, Am 19.03.26 um 10:18 AM schrieb Ryan Sleeth via ceph-users: > My understanding is a manual device add approach, while possible, removes > the benefits of orch (creating "unmanaged" devices) and should be avoided. > But would it even be possible for orch to be fed a yaml with the NVMes > operating as both a data and db device? Check if the NVMe devices support multiple namespaces: https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html This way the kernel sees two block devices for each NVMe and the orchestrator is able to provision an OSD on one and the RocksDBs for the hybrid OSDs on the other. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: tentacle 20.2.1 QE validation status
by Vallari Agrawal 20 Mar '26

20 Mar '26

nvmeof passed based on https://pulpito.ceph.com/yuriw-2026-03-06_16:22:59-nvmeof-tentacle-release-… (which does not include revert PR https://github.com/ceph/ceph/pull/67750, please let me know if we need to re-review after it's merged) thanks, Vallari On Tue, Mar 17, 2026 at 2:28 AM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > Only dashboard, rbd and krbd were approved. > > Patrick is working on https://github.com/ceph/ceph/pull/67750 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2…> > revert > > I also added tentacle-p2p run and it needs reviews. > > Dev leads, pls review/approve the run results. > > On Mon, Mar 9, 2026 at 11:44 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/74838#note-4 >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya (still running) >> nvmeof: Aviv, Vallari >> >> upgrade/squid-x: Laura >> upgrade/reef-x: Laura >> upgrade/telemetry-upgrade: Laura >> >> upgrade/tentacle-p2p: N/A >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA >> > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Re: Recommended approach for hybrid devices
by Anthony D'Atri 20 Mar '26

20 Mar '26

> I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and > 2x 2 TB enterprise NVMes. Ensure the firmware is up to date on each! > My initial plan was to split the NVMes into 5x > 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 > HDDs creating hybrid devices). My hope was that for performance, the one > NVMe data partition could then be used for 4x replicated SSD-only > metadata pools. What SSD and HDD SKUs exactly please? > The hybrid devices would be for 4+2 EC cephfs. Depending on your workload, you might find that the WAL+DB offload doesn't get you much. > > Previously I had tested this successfully on Squid using a series of manual > device adds, but on Tentacle it is seeing pre-created LVM partitions as in > use by the file system where Squid did not (ceph orch device ls ceph01). > > My understanding is a manual device add approach, while possible, removes > the benefits of orch (creating "unmanaged" devices) and should be avoided. > But would it even be possible for orch to be fed a yaml with the NVMes > operating as both a data and db device? If you specify the paths explicitly, that should work. > The obvious answer is to have a third NVMe per node operating as data for > the 4x replicated metadata pool, but I am not sure if my budget permits > that. What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.

2 3

Re: updating RBD client auth parameters
by Eugen Block 19 Mar '26

19 Mar '26

Hi, that's exactly where we landed this week with our own 11 year old cluster. We tried to access a home directory on a fresh openSUSE Leap 16 system via CephFS and failed due to an old auth cap for that user. We did migrate a couple of rbd clients over the years to the "profile rbd" syntax. But we still have some leftovers, so this is gonna take some time. I usually create new clients and compare them (like you did), then take some uncritical client and update its caps. If nobody complains, I adapt it on other clients as well. But I'm not aware of any documented update strategy, and at the same time I expect backwards compatibility for such clients. Our specific issue was an "allow r path=/" statement which MDS rejects if the user is supposed to only have access to a subdirectory. It still works with 6.4 kernel but failed with kernel 6.12. Regards, Eugen Zitat von Jan Kasprzak via ceph-users <ceph-users(a)ceph.io>: > Hello, Ceph users, > > is there a recommended way how to keep ceph users' capabilities up-to-date? > I have a many years old cluster originally based on Hammer, gradually updated > up to Tentacle, and I noticed that users created for rbd access long time ago > according to recommendations of that time differ in their capabilities: > > # ceph auth ls > [...] > client.one > key: XXX...== > caps: [mgr] allow r > caps: [mon] allow r, allow command "osd blacklist" > caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=one > client.two > key: YYY...== > caps: [mgr] allow r > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=two > client.three > key: ZZZ...== > caps: [mgr] profile rbd pool=three > caps: [mon] profile rbd > caps: [osd] profile rbd pool=three > > are the above setting equivalent on Tentacle? Can the older users > "client.one" > and "client.two" be safely updated to the same settings as the > "client.three"? > > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

updating RBD client auth parameters
by Jan Kasprzak 19 Mar '26

19 Mar '26

Hello, Ceph users, is there a recommended way how to keep ceph users' capabilities up-to-date? I have a many years old cluster originally based on Hammer, gradually updated up to Tentacle, and I noticed that users created for rbd access long time ago according to recommendations of that time differ in their capabilities: # ceph auth ls [...] client.one key: XXX...== caps: [mgr] allow r caps: [mon] allow r, allow command "osd blacklist" caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=one client.two key: YYY...== caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=two client.three key: ZZZ...== caps: [mgr] profile rbd pool=three caps: [mon] profile rbd caps: [osd] profile rbd pool=three are the above setting equivalent on Tentacle? Can the older users "client.one" and "client.two" be safely updated to the same settings as the "client.three"? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: Merging two clusters
by Anthony D'Atri 19 Mar '26

19 Mar '26

I’ve used rbd-mirror to do this sort of thing in the past. It’s straightforward with unattached volumes. I had a homebrew harness that for attached images would handle a reboot of the client with reattach. > On Mar 19, 2026, at 8:24 AM, Nico Schottelius via ceph-users <ceph-users(a)ceph.io> wrote: > > > Hello, > > I was wondering if there is any (recent, in general) method to merge to > clusters. We have several older nautilus based clusters that are > natively deployed. We also have several newer rook/kubernetes/squid > based clusters. > > In each data center we have both, alongside each other. While the > clusters are running in separate networks, we could open the firewalls > between them for a merge/migration. > > My question is, is there any reasonable way of merging an old nautilus > cluster into another existing squid cluster? > > In theory, what would be amazing is something like a fluid process in > which the newer cluster could retrieve items from the older cluster and > once it has them, decommission them from the old cluster. > > The live migration on > https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a > bit into the direction, but I am not sure if that works in our case. > > We are looking at migrating around 500 rbd images in total. > > Any hints appreciated! > > Best regards, > > Nico > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Merging two clusters
by Burkhard Linke 19 Mar '26

19 Mar '26

Hi, we used the rbd live migration in the past to split up a cluster. The processed pools were used for image and volumes in our openstack cloud. Live migration mostly works, but if the RBD image is in use, you have to add some extra effort. Cinder for examples stores the mon information in its database, and this has to be updated. I used a script to migrate a RBD image, pause the instance, adjust the database entries, and live migrate the instance itself to another host to update the qemu/libvirt information. If the image is not actively used migration is trivial; if the image is attached to a stopped virtual machine migration is also trivial, but database entries have to be updated. The rather old nautilus clusters might be problem, so if you want to try rbd live migration you need to test this first. Not sure whether the script still exists, but I can try to dig it out if necessary. Best regards, Burkhard Linke On 3/19/26 13:28, Nico Schottelius via ceph-users wrote: > Hello, > > I was wondering if there is any (recent, in general) method to merge to > clusters. We have several older nautilus based clusters that are > natively deployed. We also have several newer rook/kubernetes/squid > based clusters. > > In each data center we have both, alongside each other. While the > clusters are running in separate networks, we could open the firewalls > between them for a merge/migration. > > My question is, is there any reasonable way of merging an old nautilus > cluster into another existing squid cluster? > > In theory, what would be amazing is something like a fluid process in > which the newer cluster could retrieve items from the older cluster and > once it has them, decommission them from the old cluster. > > The live migration on > https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a > bit into the direction, but I am not sure if that works in our case. > > We are looking at migrating around 500 rbd images in total. > > Any hints appreciated! > > Best regards, > > Nico > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Merging two clusters
by Nico Schottelius 19 Mar '26

19 Mar '26

Hello, I was wondering if there is any (recent, in general) method to merge to clusters. We have several older nautilus based clusters that are natively deployed. We also have several newer rook/kubernetes/squid based clusters. In each data center we have both, alongside each other. While the clusters are running in separate networks, we could open the firewalls between them for a merge/migration. My question is, is there any reasonable way of merging an old nautilus cluster into another existing squid cluster? In theory, what would be amazing is something like a fluid process in which the newer cluster could retrieve items from the older cluster and once it has them, decommission them from the old cluster. The live migration on https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a bit into the direction, but I am not sure if that works in our case. We are looking at migrating around 500 rbd images in total. Any hints appreciated! Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Recommended approach for hybrid devices
by Ryan Sleeth 19 Mar '26

19 Mar '26

I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes. My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools. The hybrid devices would be for 4+2 EC cephfs. Previously I had tested this successfully on Squid using a series of manual device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01). My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device? The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that. -- Ryan Sleeth

1 0

Re: Cephfs kernel client bug report
by Alain Bertrand 18 Mar '26

18 Mar '26

Slava, I just put the details in the bug report https://tracker.ceph.com/issues/75609 . The kernels I was able to reproduce the issue on were the proxmox supplied 6.17.13-1-pve, and 6.14.11-5-pve, and the Pop!_OS kernel version 6.18.7-76061807-generic. The problem didn't happen on a different machine running CachyOS kernel 6.19.6-2-cachyos. All I needed to do to reproduce the problem was go to the cephfs mountpoint (/mnt/ceph) and run find -maxdepth 3 -type d -exec ls "{}/.snap/" \; which should have listed snapshots from about 1015 directories. This wasn't the smartest way to find snapshots, it was just the first one I thought of. Each time the effect is a little different. Sometimes it immediately panics, sometimes the machine just gets slow for a few minutes with lots of errors being logged before it panics. Thanks, Alain On 2026-03-18 13:07, Viacheslav Dubeyko wrote: > Hi Alain, > > The ceph-devel(a)vger.kernel.org is the email list for sharing issues in CephFS > kernel client. > > I remember that we had the fix for likewise issue recently. Could you please > share which kernel version you have? Which particular call trace do you have for > this kernel panic? How anybody can reproduce the issue? > > Thanks, > Slava. > > On Wed, 2026-03-18 at 15:17 +0530, Venky Shankar wrote: >> cc Alex/Slava (cephfs kernel driver maintainers) >> >> On Wed, Mar 18, 2026 at 3:35 AM Alain Bertrand via ceph-users >> <ceph-users(a)ceph.io> wrote: >>> Hi. >>> >>> I'm new here. I have been playing with ceph for a few years (nothing >>> dramatic, just using it in my proxmox cluster to store VM images and >>> host a cephfs with a lot of shared files). I think I found a bug in the >>> cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a >>> lot of directories quickly causes a kernel panic. I tried to create an >>> account on the bug tracker at tracker.ceph.com a week ago, but still >>> can't log in there. Do I put details here, or report somewhere else? >>> What's the next step I should take? >>> >>> >>> Thanks, >>> >>> Alain >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

Re: Cephfs kernel client bug report
by Venky Shankar 18 Mar '26

18 Mar '26

cc Alex/Slava (cephfs kernel driver maintainers) On Wed, Mar 18, 2026 at 3:35 AM Alain Bertrand via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi. > > I'm new here. I have been playing with ceph for a few years (nothing > dramatic, just using it in my proxmox cluster to store VM images and > host a cephfs with a lot of shared files). I think I found a bug in the > cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a > lot of directories quickly causes a kernel panic. I tried to create an > account on the bug tracker at tracker.ceph.com a week ago, but still > can't log in there. Do I put details here, or report somewhere else? > What's the next step I should take? > > > Thanks, > > Alain > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Cheers, Venky

1 0

Restricting subuser to its bucket
by Rok Jaklič 18 Mar '26

18 Mar '26

Hi, I am trying to restrict subuser to its bucket (so he/she can work only on a bucket which has permission) with disabling ACLs and setting policy on a bucket: Using ceph version 19.2.2 bucket1-policy.json contains { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::tenant1:user/parent1:user1" }, "Action": "s3:*", "Resource": [ "arn:aws:s3:::bucket1", "arn:aws:s3:::bucket1/*" ] } ] } --- echo "lala" > test.txt export RGW_ENDPOINT="" radosgw-admin user create --tenant=tenant1 --uid=parent1 --key-type=s3 --display-name="tenant1:parent1" for sub in user1 user2 user3; do radosgw-admin subuser create --tenant=tenant1 --uid=parent1 --subuser=$sub --key-type=s3 --access=full; done export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user1").access_key') export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user1").secret_key') aws s3api create-bucket --bucket bucket1 --object-ownership BucketOwnerEnforced --endpoint-url $RGW_ENDPOINT aws s3api put-bucket-policy --bucket bucket1 --policy file://bucket1-policy.json --endpoint-url $RGW_ENDPOINT aws s3api put-public-access-block --bucket bucket1 --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true --endpoint-url $RGW_ENDPOINT aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT # verify export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user2").access_key') export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" --tenant="tenant1" | jq .keys | jq -r '.[] | select(.user=="tenant1$parent1:user2").secret_key') aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT --- Script stops at aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT with an error like: upload failed: ./test.txt to s3://bucket1/test.txt argument of type 'NoneType' is not iterable If I comment the line with aws s3api put-public-access-block... I can successfully upload some file, but I can also see the contents of bucket1 with user2, which is not what I want. Any ideas how would I restrict the subuser to its bucket only? I was actually able to restrict subuser to its bucket by explicitly denying access to bucket with following policy: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/AFNERFMNSC… But it doesn't feel the right way. Wouldn't be better to "restrict" everything for everyone but its user in the first place and then allow others to access? Kind regards, Rok

1 0

Re: Cephfs kernel client bug report
by Wannes Smet 18 Mar '26

18 Mar '26

Hi Alain, You could talk to Anthony Middleton. Usually he can speed up your bug account approval request. In case you're active on Slack, you can find him there. Wannes ________________________________ From: MARTEL Arnaud via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 18, 2026 08:58 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Cephfs kernel client bug report Hi Alain, It seems to be related to this kernel bug: https: //urldefense. com/v3/__https: //bugzilla. kernel. org/show_bug. cgi?id=220807__;!!FtrhtPsWDhZ6tw!GXjnqm57m9uDNp3YBpMOJiB1tmvYHyMzZ0h2Fppd1HSPnGMLKlU3L3M--WSNMLIMhoDxCFX_qMTW7Rkc$[bugzilla[. ]kernel[. ]org] ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Alain, It seems to be related to this kernel bug: https://urldefense.com/v3/__https://bugzilla.kernel.org/show_bug.cgi?id=220… Upgrading your kernel should be a solution on Debian 13 (kernel 6.12.73-1 is OK) Arnaud Le 17/03/2026 23:07, « Alain Bertrand via ceph-users » <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> a écrit : Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephfs kernel client bug report
by MARTEL Arnaud 18 Mar '26

18 Mar '26

Hi Alain, It seems to be related to this kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=220807 Upgrading your kernel should be a solution on Debian 13 (kernel 6.12.73-1 is OK) Arnaud Le 17/03/2026 23:07, « Alain Bertrand via ceph-users » <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> a écrit : Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

Windows CEPH driver Re: Re: CSC minutes 2026-03-16
by Takeshi Nishimura 18 Mar '26

18 Mar '26

On Tue, Mar 17, 2026 at 7:46 PM Martin Wege via Dev <dev(a)ceph.io> wrote: > > On Mon, Mar 16, 2026 at 6:46 PM Takeshi Nishimura via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > > > > 2026/03/16 > > > > > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe > > ... > > > Ceph on Windows discussion > > > Weekly meeting was set up on the Community Calendar on Tuesdays > > > Alexandre Jardon, Andy Herron involved > > > > > > Will come back to CSC next week to ask opinion on plan of action > > > Goal will be to improve understanding/direction around Windows > > > > Could you please consider getting Roland Mainz > > <roland.mainz(a)nrubsig.org> involved? > > > > Mr Mainz wrote ms-nfs41-client (link: > > https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… > > and https://github.com/kofemann/ms-nfs41-client) and should have > > sufficient expertise to provide comments. > > ms-nfs41-client is a Windows NFSv4.2 filesystem client. How should it > be able to help with Ceph? I explained that already: 1. IMO Mr Mainz has the expertise to comment on the subject of Windows kernel filesystem drivers 2. This specific driver has a kernel part and a userland daemon, and can be used as a template for a Windows CEPH driver. Replace the NFS parts with CEPH parts. 3. You wrote in https://www.spinics.net/lists/dev-ceph/msg06898.html that one of the better choices for a CEPH Windows driver is a RDBSS driver. ms-nfs41-client uses RDBSS. -- Internationalization&localization dev / 大阪大学 Takeshi Nishimura <takeshi.nishimura.linux(a)gmail.com>

1 0

Re: Cephfs kernel client bug report
by John Mulligan 17 Mar '26

17 Mar '26

On Tuesday, March 17, 2026 6:09:29 PM Eastern Daylight Time Alain Bertrand via ceph-users wrote: > Hi. > > I'm new here. I have been playing with ceph for a few years (nothing > dramatic, just using it in my proxmox cluster to store VM images and > host a cephfs with a lot of shared files). I think I found a bug in the > cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a > lot of directories quickly causes a kernel panic. I tried to create an > account on the bug tracker at tracker.ceph.com a week ago, but still > can't log in there. Do I put details here, or report somewhere else? > What's the next step I should take? > Tracker is the right place. Tracker approval requires manual review and unfortunately that makes the sign-up process very slow. You can join ceph slack and ask for updates the #sepia channel - for now I will do that so you don't have to jump through more hoops right now.

1 0

Cephfs kernel client bug report
by Alain Bertrand 17 Mar '26

17 Mar '26

Hi. I'm new here. I have been playing with ceph for a few years (nothing dramatic, just using it in my proxmox cluster to store VM images and host a cephfs with a lot of shared files). I think I found a bug in the cephfs client in the Linux kernel. Listing snapshots (ls .snap) in a lot of directories quickly causes a kernel panic. I tried to create an account on the bug tracker at tracker.ceph.com a week ago, but still can't log in there. Do I put details here, or report somewhere else? What's the next step I should take? Thanks, Alain

1 0

Re: CSC minutes 2026-03-16
by Martin Wege 17 Mar '26

17 Mar '26

On Mon, Mar 16, 2026 at 6:46 PM Takeshi Nishimura via ceph-users <ceph-users(a)ceph.io> wrote: > > On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > > 2026/03/16 > > > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe > ... > > Ceph on Windows discussion > > Weekly meeting was set up on the Community Calendar on Tuesdays > > Alexandre Jardon, Andy Herron involved > > > > Will come back to CSC next week to ask opinion on plan of action > > Goal will be to improve understanding/direction around Windows > > Could you please consider getting Roland Mainz > <roland.mainz(a)nrubsig.org> involved? > > Mr Mainz wrote ms-nfs41-client (link: > https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… > and https://github.com/kofemann/ms-nfs41-client) and should have > sufficient expertise to provide comments. ms-nfs41-client is a Windows NFSv4.2 filesystem client. How should it be able to help with Ceph? Thanks, Martin

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 17 Mar '26

17 Mar '26

This point release is ready We need to finish reviewing and approving release notes and add notable changes. https://github.com/ceph/ceph/pull/67746 On Tue, Mar 10, 2026 at 8:53 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Patrick approved fs and we will continue with this release > > On Wed, Feb 25, 2026 at 7:43 AM Patrick Donnelly <pdonnell(a)redhat.com> > wrote: > > > The `fs` and `powercycle` suites will need rerun with [1]. I believe > > you're already doing that but I'm documenting that for the list. > > > > I'm going to look into the upgrade issues today [2]. > > > > [1] https://github.com/ceph/ceph/pull/67495 > > [2] https://tracker.ceph.com/issues/75034 > > > > On Mon, Feb 23, 2026 at 4:29 PM Yuri Weinstein via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > > > Patrick, > > > > > > The only outstanding issues are the FS approval and upgrade issues, > > > reported by Laura. > > > > > > What do you suggest we do next here? > > > > > > On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > > > > > > > orch approved. Run looks ugly due to us no having a proper ignorelist > > for > > > > the orch suite on reef, but the actual results are okay > > > > > > > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > > > > ceph-users(a)ceph.io> wrote: > > > > > > > >> Rerun scheduled for krbd > > > >> > > > >> Still waiting for replies from dev leads > > > >> > > > >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> > > wrote: > > > >> > > > >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein < > > yweinste(a)redhat.com> > > > >> > wrote: > > > >> > > > > > >> > > Note: this is an attempt to test and publish post EOL reef > point > > > >> release > > > >> > > > > > >> > > Details of this release are summarized here: > > > >> > > > > > >> > > https://tracker.ceph.com/issues/73906#note-1 > > > >> < > > > https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F… > > > > > > >> > > Release Notes - TBD > > > >> > > LRC upgrade - N/A > > > >> > > > > > >> > > distro => jammy focal centos9 windows bookworm > > > >> > > > > > >> > > Seeking approvals/reviews for: > > > >> > > > > > >> > > smoke - Laura approved? > > > >> > > > > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > > >> approved? > > > >> > > (rados is also running on build 2 == build 1 + > > > >> > > https://github.com/ceph/ceph/pull/66056 > > > >> < > > > https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2… > > > > > > >> per Igor's request) > > > >> > > > > > >> > > rgw - Adam E approved? > > > >> > > > > > >> > > fs - Venky approved? > > > >> > > > > > >> > > orch - Adam King approved? > > > >> > > > > > >> > > rbd - Ilya approved? > > > >> > > krbd - Ilya approved? > > > >> > > > > >> > rbd approved, krbd needs a rerun as all jobs failed while > > > >> > going through cephlab.yml playbook (not related to RBD): > > > >> > > > > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > type, > > > >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > > > >> > codepage or helper program, or other error.'] > > > >> > > > > >> > Thanks, > > > >> > > > > >> > Ilya > > > >> > > > > >> > > > > >> _______________________________________________ > > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > >> > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > -- > > Patrick Donnelly, Ph.D. > > He / Him / His > > Red Hat Partner Engineer > > IBM, Inc. > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

hi burkhard, i found that way and build my own script for that. but thanks, its good to get some experience. sometimes not easy, to find the right way ... had some issues with rbd|cephfs-mirroring. so, very good information for that money ;) i also used that for the samba dfree command script. its now working fine. little notice ... map a network drive and there is no "X GB of Y GB free" line. right click, rename the network drive and voila ... there it is ... funny. regards, ronny Am 2026-03-17 12:40, schrieb Burkhard Linke via ceph-users: > Hi, > > > just my 0.02 euro: > > > if you want to collect usage information on the cluster, you can use > the recursive stats feature in cephfs: > > $ getfattr -n ceph.dir.rbytes /vol/adm > getfattr: Removing leading '/' from absolute path names > # file: vol/adm > ceph.dir.rbytes="1199125900626" > > $ getfattr -n ceph.dir.rfiles /vol/adm > getfattr: Removing leading '/' from absolute path names > # file: vol/adm > ceph.dir.rfiles="309017" > > > We use a script to generate the corresponding values in a prometheus > compatible format, and use script-exporter for invoking the script > remotely. Results are stored in prometheus/thanos and visualized with > grafana. > > > This will not show correct values for users, but you can also export > the current quotas (also just xattrs), and use prometheus or grafana > alerting if certain thresholds are reached. > > > Best regards, > > Burkhard > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Cephadm staggered upgrade --limit parameter not always applied
by Eugen Block 17 Mar '26

17 Mar '26

Hi, I did a quick search on tracker but couldn't find anything related. A customer reported this, and I can confirm the behaviour on a lab cluster. I usually perform a staggered upgrade with --daemon-types and --limit, but not for all daemon types. So I haven't stumbled across this yet myself, but our customer did (I can reproduce with --daemon-types as well). They upgraded from latest Reef to latest Squid and reported that despite providing the --limit parameter, the mons were all upgraded. So I tried to reproduce, and the behaviour is not really clear to me, I'll try to clarify. # Start with MGRs reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 Upgrading MGRs with limit works, but it doesn't reflect in the MGR log. Usually, I expect a line like this: ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade But there is no such line in the logs. Then I upgrade the rest of the MGRs. # Continue with MONs reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. And again, no such line in the log (Hit upgrade limit). What I noticed was a MGR respawn after the first MON had been upgraded successfully. Maybe some state of the upgrade progress gets lost during the respawn? I then upgraded the remaining MON. Then ceph-crash is upgraded successfully. # Upgrade OSD reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 And this is the first service that actually reports the upgrade limit: 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Complete! 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Complete! This is really irritating and inconsistent: If the orchestrator does honor --limit with other services than OSD, why isn't that visible in the logs? And what's with the MONs? Why 2 out of 3? Any pointers appreciated! Not sure which Ceph versions might be affected by this, I'll try out a couple more upgrade paths. Thanks, Eugen

1 2

Re: cephfs quota samba export
by Burkhard Linke 17 Mar '26

17 Mar '26

Hi, just my 0.02 euro: if you want to collect usage information on the cluster, you can use the recursive stats feature in cephfs: $ getfattr -n ceph.dir.rbytes /vol/adm getfattr: Removing leading '/' from absolute path names # file: vol/adm ceph.dir.rbytes="1199125900626" $ getfattr -n ceph.dir.rfiles /vol/adm getfattr: Removing leading '/' from absolute path names # file: vol/adm ceph.dir.rfiles="309017" We use a script to generate the corresponding values in a prometheus compatible format, and use script-exporter for invoking the script remotely. Results are stored in prometheus/thanos and visualized with grafana. This will not show correct values for users, but you can also export the current quotas (also just xattrs), and use prometheus or grafana alerting if certain thresholds are reached. Best regards, Burkhard

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

only windows explorer will not show the space value i see name, letter and an usage diagram. mouse over shows me the correct value. but i miss "90mb free of 120mb". if you do not have an spontaneous idea ... we can close here its not samba list. thanks to all the people here. great to find community help from real users. regards, ronny Am 2026-03-17 07:45, schrieb Nemesiz via ceph-users: > I recommend to use external script or program for storage report. > > "dfree" https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#DFREECOMM… > > > On 2026-03-17 08:27, ronny.lippold via ceph-users wrote: >> many thanks ... did not found something in the archive, will search >> again. >> >> i found out, it looks like an samba issue >> >> i set up: >> data/users/user1 75M >> >> i have 2 shares in samba >> users -> path=data/users >> user1 -> path=data/users/user1 >> >> if i mount //server/users/user1 -> quota is not shown >> (//server/users/user1 5.5T 3.8G 5.5T 1% /mnt) >> >> if i mount the //server/user1 -> quota is shown correctly >> (//server/user1 72M 36M 36M 50% /mnt) >> >> looks bad to me, don't see an solution. >> >> >> regards, >> ronny >> >> >> >> Am 2026-03-16 16:05, schrieb Chris Palmer: >>> My next question was whether you were trying to apply quotas at >>> multiple points between the root and the target, and it appears you >>> are. IIRC there were discussions about that on this list some years >>> ago. I'm not sure that it is supported, and I don't really see how it >>> could be. However I'm not an expert in this area, so perhaps someone >>> else can chime in. >>> >>> On 16/03/2026 14:22, ronny.lippold wrote: >>>> update, burns my head down :) >>>> >>>> first level is working fine, under >>>> /mnt/pve/cephfs/shares/FOLDER/data/ >>>> server1 >>>> server2 >>>> users >>>> (in samba and smb client) >>>> >>>> one step down, not >>>> /mnt/pve/cephfs/shares/FOLDER/data/users >>>> username >>>> >>>> if i mount via ceph on the samba server >>>> /mnt/pve/cephfs/shares/FOLDER/data/users/username >>>> i see the correct quota size >>>> SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M >>>> 0% /srv/samba/data/username >>>> >>>> if i mount this via smb client no >>>> //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 >>>> >>>> >>>> ronny >>>> >>>> >>>> Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: >>>>> Hi Ronny >>>>> >>>>> I use the very straightforward method of having a linux VM that >>>>> kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I >>>>> apply an 8TB quota to one of the top-level directories >>>>> /mnt/cephfs/data2 (not the root of /mnt/cephfs as I mistakenly said >>>>> earlier!). Samba then exports /mnt/cephfs/data2/x, >>>>> /mnt/cephfs/data2/y etc just via the VM's directory tree. Each of >>>>> the exports (some with different access permissions etc specified >>>>> in samba) all correctly share the quota set on data2. >>>>> >>>>> This is a low-usage cluster, especially via samba, and has always >>>>> just worked. So I've never had any reason to experiment with vfs >>>>> etc. >>>>> >>>>> Regards, Chris >>>>> >>>>> On 16/03/2026 11:07, ronny.lippold wrote: >>>>>> hi chris and thanks for answer. >>>>>> >>>>>> do you use samba vfs ceph/ceph_new or mount cephfs on samba server >>>>>> and export that? >>>>>> what do you mean, "cephfs root though" >>>>>> >>>>>> i would like to have something like that >>>>>> >>>>>> data/users/user1 10g >>>>>> data/users/user2 5g >>>>>> data/users/user3 20g >>>>>> data/users/user4 10g >>>>>> data/share1 100g >>>>>> data/share2 200g >>>>>> ... >>>>>> >>>>>> regards, >>>>>> ronny >>>>>> >>>>>> >>>>>> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>>>>>> I do use cephfs exported via samba. My quota is simply done at >>>>>>> the cephfs root though. That works fine with Win10 & Win11 hosts >>>>>>> - they see the correct total/used/available space. I haven't >>>>>>> tried multiple quotas or setting further down the tree though. >>>>>>> >>>>>>> Regards, Chris >>>>>>> >>>>>>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>>>>>> hi, hope you can help me. >>>>>>>> >>>>>>>> i like to export an cephfs via samba an ctdb. >>>>>>>> its all working fine via kernel cephfs client. >>>>>>>> >>>>>>>> i set now quota to the users home dir. >>>>>>>> quota also working fine. >>>>>>>> >>>>>>>> but now, the problem ... on the client machine, i do not see the >>>>>>>> quota space. >>>>>>>> mostly, i see the complete cluster, or on windows i see nothing >>>>>>>> about disk usage. >>>>>>>> >>>>>>>> it would be important for us and the user, to se the remaining >>>>>>>> space. >>>>>>>> >>>>>>>> samba vfs ceph and ceph_new would not work ... i always got: >>>>>>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>>>>>> fsname='XXX' No such file or directorymake_connection_snum: >>>>>>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file >>>>>>>> or directory >>>>>>>> >>>>>>>> does anybody use cephfs with samba an quota? >>>>>>>> running out of ideas, so an help is welcome. >>>>>>>> >>>>>>>> regards, >>>>>>>> ronny >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

great, thanks ... danke :) what a "hack" ... i love it. dfree command is working fine. have a great day, ronny Am 2026-03-17 07:45, schrieb Nemesiz via ceph-users: > I recommend to use external script or program for storage report. > > "dfree" https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#DFREECOMM… > > > On 2026-03-17 08:27, ronny.lippold via ceph-users wrote: >> many thanks ... did not found something in the archive, will search >> again. >> >> i found out, it looks like an samba issue >> >> i set up: >> data/users/user1 75M >> >> i have 2 shares in samba >> users -> path=data/users >> user1 -> path=data/users/user1 >> >> if i mount //server/users/user1 -> quota is not shown >> (//server/users/user1 5.5T 3.8G 5.5T 1% /mnt) >> >> if i mount the //server/user1 -> quota is shown correctly >> (//server/user1 72M 36M 36M 50% /mnt) >> >> looks bad to me, don't see an solution. >> >> >> regards, >> ronny >> >> >> >> Am 2026-03-16 16:05, schrieb Chris Palmer: >>> My next question was whether you were trying to apply quotas at >>> multiple points between the root and the target, and it appears you >>> are. IIRC there were discussions about that on this list some years >>> ago. I'm not sure that it is supported, and I don't really see how it >>> could be. However I'm not an expert in this area, so perhaps someone >>> else can chime in. >>> >>> On 16/03/2026 14:22, ronny.lippold wrote: >>>> update, burns my head down :) >>>> >>>> first level is working fine, under >>>> /mnt/pve/cephfs/shares/FOLDER/data/ >>>> server1 >>>> server2 >>>> users >>>> (in samba and smb client) >>>> >>>> one step down, not >>>> /mnt/pve/cephfs/shares/FOLDER/data/users >>>> username >>>> >>>> if i mount via ceph on the samba server >>>> /mnt/pve/cephfs/shares/FOLDER/data/users/username >>>> i see the correct quota size >>>> SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M >>>> 0% /srv/samba/data/username >>>> >>>> if i mount this via smb client no >>>> //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 >>>> >>>> >>>> ronny >>>> >>>> >>>> Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: >>>>> Hi Ronny >>>>> >>>>> I use the very straightforward method of having a linux VM that >>>>> kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I >>>>> apply an 8TB quota to one of the top-level directories >>>>> /mnt/cephfs/data2 (not the root of /mnt/cephfs as I mistakenly said >>>>> earlier!). Samba then exports /mnt/cephfs/data2/x, >>>>> /mnt/cephfs/data2/y etc just via the VM's directory tree. Each of >>>>> the exports (some with different access permissions etc specified >>>>> in samba) all correctly share the quota set on data2. >>>>> >>>>> This is a low-usage cluster, especially via samba, and has always >>>>> just worked. So I've never had any reason to experiment with vfs >>>>> etc. >>>>> >>>>> Regards, Chris >>>>> >>>>> On 16/03/2026 11:07, ronny.lippold wrote: >>>>>> hi chris and thanks for answer. >>>>>> >>>>>> do you use samba vfs ceph/ceph_new or mount cephfs on samba server >>>>>> and export that? >>>>>> what do you mean, "cephfs root though" >>>>>> >>>>>> i would like to have something like that >>>>>> >>>>>> data/users/user1 10g >>>>>> data/users/user2 5g >>>>>> data/users/user3 20g >>>>>> data/users/user4 10g >>>>>> data/share1 100g >>>>>> data/share2 200g >>>>>> ... >>>>>> >>>>>> regards, >>>>>> ronny >>>>>> >>>>>> >>>>>> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>>>>>> I do use cephfs exported via samba. My quota is simply done at >>>>>>> the cephfs root though. That works fine with Win10 & Win11 hosts >>>>>>> - they see the correct total/used/available space. I haven't >>>>>>> tried multiple quotas or setting further down the tree though. >>>>>>> >>>>>>> Regards, Chris >>>>>>> >>>>>>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>>>>>> hi, hope you can help me. >>>>>>>> >>>>>>>> i like to export an cephfs via samba an ctdb. >>>>>>>> its all working fine via kernel cephfs client. >>>>>>>> >>>>>>>> i set now quota to the users home dir. >>>>>>>> quota also working fine. >>>>>>>> >>>>>>>> but now, the problem ... on the client machine, i do not see the >>>>>>>> quota space. >>>>>>>> mostly, i see the complete cluster, or on windows i see nothing >>>>>>>> about disk usage. >>>>>>>> >>>>>>>> it would be important for us and the user, to se the remaining >>>>>>>> space. >>>>>>>> >>>>>>>> samba vfs ceph and ceph_new would not work ... i always got: >>>>>>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>>>>>> fsname='XXX' No such file or directorymake_connection_snum: >>>>>>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file >>>>>>>> or directory >>>>>>>> >>>>>>>> does anybody use cephfs with samba an quota? >>>>>>>> running out of ideas, so an help is welcome. >>>>>>>> >>>>>>>> regards, >>>>>>>> ronny >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Re: problem with mgr prometheus module
by JOUSSE Adrien 17 Mar '26

17 Mar '26

Hello, Quite an hold thread, but maybe this will help someone. We had the same issue on our Ceph cluster. It all came down to an erasure code profile with a missing setting. In the logs of ceph-mgr (journalctl -xru ceph-mgr@<host>.service), we saw the following error: KeyError: 'm' description = f"ec:{profile['k']}+{profile['m']}" in /usr/share/ceph/mgr/prometheus/module.py. One of our erasure code profile was missing the setting "m" (which is the number of parity chunks for a chunk of data, if I understood correctly). You can see your erasure code profile using : ceph osd erasure-code-profile ls And the settings of each code profile with : ceph osd erasure-code-profile get <name> We deployed a simple fix by editing the module.py file to handle this scenario. Replace (around l.1250) if profile: description = f"ec:{profile['k']}+{profile['m']}" With if profile: if profile.get("m", None) is not None: description = f"ec:{profile['k']}+{profile['m']}" else: description = f"ec:{profile['k']}" Adrien [Site du CEA]<https://www.cea.fr/> Adrien JOUSSE Ingénieur systèmes Unix Direction de la Recherche Fondamentale (DRF) DRF/IRFU/DEDIP//LIS Centre de Saclay [Facebook]<https://fr-fr.facebook.com/cea.pageofficielle> [Youtube] <https://www.youtube.com/channel/UCpTMWY8NRQby8zXUKhHznfg> [Twitter] <https://twitter.com/CEA_Officiel> [LinkedIn] <https://fr.linkedin.com/company/cea> [Instagram] <https://www.instagram.com/cea_officiel/>

1 0

Re: cephfs quota samba export
by Nemesiz 17 Mar '26

17 Mar '26

I recommend to use external script or program for storage report. "dfree" https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#DFREECOMM… On 2026-03-17 08:27, ronny.lippold via ceph-users wrote: > many thanks ... did not found something in the archive, will search > again. > > i found out, it looks like an samba issue > > i set up: > data/users/user1 75M > > i have 2 shares in samba > users -> path=data/users > user1 -> path=data/users/user1 > > if i mount //server/users/user1 -> quota is not shown > (//server/users/user1 5.5T 3.8G 5.5T 1% /mnt) > > if i mount the //server/user1 -> quota is shown correctly > (//server/user1 72M 36M 36M 50% /mnt) > > looks bad to me, don't see an solution. > > > regards, > ronny > > > > Am 2026-03-16 16:05, schrieb Chris Palmer: >> My next question was whether you were trying to apply quotas at >> multiple points between the root and the target, and it appears you >> are. IIRC there were discussions about that on this list some years >> ago. I'm not sure that it is supported, and I don't really see how it >> could be. However I'm not an expert in this area, so perhaps someone >> else can chime in. >> >> On 16/03/2026 14:22, ronny.lippold wrote: >>> update, burns my head down :) >>> >>> first level is working fine, under >>> /mnt/pve/cephfs/shares/FOLDER/data/ >>> server1 >>> server2 >>> users >>> (in samba and smb client) >>> >>> one step down, not >>> /mnt/pve/cephfs/shares/FOLDER/data/users >>> username >>> >>> if i mount via ceph on the samba server >>> /mnt/pve/cephfs/shares/FOLDER/data/users/username >>> i see the correct quota size >>> SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M >>> 0% /srv/samba/data/username >>> >>> if i mount this via smb client no >>> //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 >>> >>> >>> ronny >>> >>> >>> Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: >>>> Hi Ronny >>>> >>>> I use the very straightforward method of having a linux VM that >>>> kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I >>>> apply an 8TB quota to one of the top-level directories >>>> /mnt/cephfs/data2 (not the root of /mnt/cephfs as I mistakenly said >>>> earlier!). Samba then exports /mnt/cephfs/data2/x, >>>> /mnt/cephfs/data2/y etc just via the VM's directory tree. Each of >>>> the exports (some with different access permissions etc specified >>>> in samba) all correctly share the quota set on data2. >>>> >>>> This is a low-usage cluster, especially via samba, and has always >>>> just worked. So I've never had any reason to experiment with vfs etc. >>>> >>>> Regards, Chris >>>> >>>> On 16/03/2026 11:07, ronny.lippold wrote: >>>>> hi chris and thanks for answer. >>>>> >>>>> do you use samba vfs ceph/ceph_new or mount cephfs on samba server >>>>> and export that? >>>>> what do you mean, "cephfs root though" >>>>> >>>>> i would like to have something like that >>>>> >>>>> data/users/user1 10g >>>>> data/users/user2 5g >>>>> data/users/user3 20g >>>>> data/users/user4 10g >>>>> data/share1 100g >>>>> data/share2 200g >>>>> ... >>>>> >>>>> regards, >>>>> ronny >>>>> >>>>> >>>>> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>>>>> I do use cephfs exported via samba. My quota is simply done at >>>>>> the cephfs root though. That works fine with Win10 & Win11 hosts >>>>>> - they see the correct total/used/available space. I haven't >>>>>> tried multiple quotas or setting further down the tree though. >>>>>> >>>>>> Regards, Chris >>>>>> >>>>>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>>>>> hi, hope you can help me. >>>>>>> >>>>>>> i like to export an cephfs via samba an ctdb. >>>>>>> its all working fine via kernel cephfs client. >>>>>>> >>>>>>> i set now quota to the users home dir. >>>>>>> quota also working fine. >>>>>>> >>>>>>> but now, the problem ... on the client machine, i do not see the >>>>>>> quota space. >>>>>>> mostly, i see the complete cluster, or on windows i see nothing >>>>>>> about disk usage. >>>>>>> >>>>>>> it would be important for us and the user, to se the remaining >>>>>>> space. >>>>>>> >>>>>>> samba vfs ceph and ceph_new would not work ... i always got: >>>>>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>>>>> fsname='XXX' No such file or directorymake_connection_snum: >>>>>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file >>>>>>> or directory >>>>>>> >>>>>>> does anybody use cephfs with samba an quota? >>>>>>> running out of ideas, so an help is welcome. >>>>>>> >>>>>>> regards, >>>>>>> ronny >>>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

update, burns my head down :) first level is working fine, under /mnt/pve/cephfs/shares/FOLDER/data/ server1 server2 users (in samba and smb client) one step down, not /mnt/pve/cephfs/shares/FOLDER/data/users username if i mount via ceph on the samba server /mnt/pve/cephfs/shares/FOLDER/data/users/username i see the correct quota size SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M 0% /srv/samba/data/username if i mount this via smb client no //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 ronny Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: > Hi Ronny > > I use the very straightforward method of having a linux VM that > kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I apply > an 8TB quota to one of the top-level directories /mnt/cephfs/data2 (not > the root of /mnt/cephfs as I mistakenly said earlier!). Samba then > exports /mnt/cephfs/data2/x, /mnt/cephfs/data2/y etc just via the VM's > directory tree. Each of the exports (some with different access > permissions etc specified in samba) all correctly share the quota set > on data2. > > This is a low-usage cluster, especially via samba, and has always just > worked. So I've never had any reason to experiment with vfs etc. > > Regards, Chris > > On 16/03/2026 11:07, ronny.lippold wrote: >> hi chris and thanks for answer. >> >> do you use samba vfs ceph/ceph_new or mount cephfs on samba server and >> export that? >> what do you mean, "cephfs root though" >> >> i would like to have something like that >> >> data/users/user1 10g >> data/users/user2 5g >> data/users/user3 20g >> data/users/user4 10g >> data/share1 100g >> data/share2 200g >> ... >> >> regards, >> ronny >> >> >> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>> I do use cephfs exported via samba. My quota is simply done at the >>> cephfs root though. That works fine with Win10 & Win11 hosts - they >>> see the correct total/used/available space. I haven't tried multiple >>> quotas or setting further down the tree though. >>> >>> Regards, Chris >>> >>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>> hi, hope you can help me. >>>> >>>> i like to export an cephfs via samba an ctdb. >>>> its all working fine via kernel cephfs client. >>>> >>>> i set now quota to the users home dir. >>>> quota also working fine. >>>> >>>> but now, the problem ... on the client machine, i do not see the >>>> quota space. >>>> mostly, i see the complete cluster, or on windows i see nothing >>>> about disk usage. >>>> >>>> it would be important for us and the user, to se the remaining >>>> space. >>>> >>>> samba vfs ceph and ceph_new would not work ... i always got: >>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>> fsname='XXX' No such file or directorymake_connection_snum: >>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file or >>>> directory >>>> >>>> does anybody use cephfs with samba an quota? >>>> running out of ideas, so an help is welcome. >>>> >>>> regards, >>>> ronny >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: CSC minutes 2026-03-16
by Takeshi Nishimura 16 Mar '26

16 Mar '26

On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > 2026/03/16 > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe ... > Ceph on Windows discussion > Weekly meeting was set up on the Community Calendar on Tuesdays > Alexandre Jardon, Andy Herron involved > > Will come back to CSC next week to ask opinion on plan of action > Goal will be to improve understanding/direction around Windows Could you please consider getting Roland Mainz <roland.mainz(a)nrubsig.org> involved? Mr Mainz wrote ms-nfs41-client (link: https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… and https://github.com/kofemann/ms-nfs41-client) and should have sufficient expertise to provide comments. -- Internationalization&localization dev / 大阪大学 Takeshi Nishimura <takeshi.nishimura.linux(a)gmail.com>

1 0

Ceph Days London CFP in now open!
by Anthony Middleton 16 Mar '26

16 Mar '26

Happy Monday, Ceph Friends! The Call for Proposals is now open for Ceph Days London 2026! 🎡💂🏼‍♂️🇬🇧 Do you have a story to share? We’re looking for talks on topics like: - Ceph operations, management, and development - New and proposed Ceph features, development status - Ceph development roadmap - Best practices - Ceph use-cases, solution architectures, and user experiences - Ceph performance and optimization - Platform Integrations - Kubernetes, OpenShift - OpenStack (Cinder, Manila, etc.) - Spark - Multi-site and multi-cluster data services - Persistent memory, ZNS SSDs, SMR HDDs, DPUs, and other new hardware technologies - Storage management, monitoring, and deployment automation - Experiences deploying and operating Ceph in production and/or at scale - Small-scale or edge deployments - Long-term, archival storage - Data compression, deduplication, and storage optimization - Developer processes, tools, challenges - Ceph testing infrastructure, tools - Ceph community issues, outreach, and project governance - Ceph documentation, training, and learner experience We want to hear from you, whether you run Ceph in production or are building the next generation of storage innovation. *Submit your talk by April 24 here: https://docs.google.com/forms/d/e/1FAIpQLSeJaOGyO0FfD9MNyCwoFdj3MzBKbRcQJZe… <https://lnkd.in/gca8mJu6>* Best, -- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

The ceph-deploy repository is now archived
by Patrick Donnelly 16 Mar '26

16 Mar '26

At today's Ceph Steering Committee meeting, it was agreed that the ceph-deploy repository [1] be archived because it is no longer maintained. Additionally, expect the removal of all references to ceph-deploy from ceph/ceph.git in the coming days. [1] https://github.com/ceph/ceph-deploy -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

CSC minutes 2026-03-16
by Laura Flores 16 Mar '26

16 Mar '26

2026/03/16 - [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe - why? this can be a solution to some technical decision making deadlocks we have from time to time - came from this discussion: https://ceph-storage.slack.com/archives/C05R975114L/p1772629034963979 - next step: feedback pls -> get the text in shape to vote on -> csc vote -> merge. - [bump to next week; Dan on PTO] - Leads, vote in the meantime - [Anthony Middleton] CSC Election - Vote on adding a removal policy to the governance, PR - https://github.com/ceph/ceph/pull/67764 - Vote on the nominated CSC members to become Co-Chairs - Need guidance on when to start the election process - Ballots will be emailed this week (likely Wed.) - People will have time to review and ask questions - Ceph on Windows discussion - Weekly meeting was set up on the Community Calendar on Tuesdays - Alexandre Jardon, Andy Herron involved - Will come back to CSC next week to ask opinion on plan of action - Goal will be to improve understanding/direction around Windows - [Patrick] Shall we archive ceph/ceph-deploy and other vestiges? - Removed from upstream qa: https://github.com/ceph/ceph/pull/67682 - e.g. references in teuthology.git - what else? (Documentation) - Patrick will alert the mailing list - Release updates - Umbrella - working on merging kickoff - ubuntu 22.04 and rocky 10 work ongoing - Tentacle - Blocker found (nvmeof feature bit) - reverting problem commit - Need p2p tests (Yuri has a draft) - Need containerization tests - Squid - Merge window is still open - Reef - Packages/containers are done - TODO: release notes -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: cephfs quota samba export
by ronny.lippold 16 Mar '26

16 Mar '26

i can confirm ... as you told, its working the "top" level accepts the quota vm-prod-fsusr-01:~# scripts/ceph_quota_report.sh Directory Used Quota Percent -------------------------------------------------------------------------------- users 3.8GiB 1.2GiB 333% users/1 0B 1.1GiB 0% users/1/2 0B 954MiB 0% users/1/2/3 0B unlimited N/A users/1/2/3/4 0B unlimited N/A but my samba mounts: //sambasrv/1 1.2G 1.2G 0 100% /mnt3 //sambasrv/users/1 1.2G 1.2G 0 100% /mnt2 //sambasrv/users/1/2 1.2G 1.2G 0 100% /mnt i dont know, how i can set different quota in one directories subdirectory like: data/users/user1 10g data/users/user2 5g data/users/user3 20g data/users/user4 10g regards, ronny Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: > Hi Ronny > > I use the very straightforward method of having a linux VM that > kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I apply > an 8TB quota to one of the top-level directories /mnt/cephfs/data2 (not > the root of /mnt/cephfs as I mistakenly said earlier!). Samba then > exports /mnt/cephfs/data2/x, /mnt/cephfs/data2/y etc just via the VM's > directory tree. Each of the exports (some with different access > permissions etc specified in samba) all correctly share the quota set > on data2. > > This is a low-usage cluster, especially via samba, and has always just > worked. So I've never had any reason to experiment with vfs etc. > > Regards, Chris > > On 16/03/2026 11:07, ronny.lippold wrote: >> hi chris and thanks for answer. >> >> do you use samba vfs ceph/ceph_new or mount cephfs on samba server and >> export that? >> what do you mean, "cephfs root though" >> >> i would like to have something like that >> >> data/users/user1 10g >> data/users/user2 5g >> data/users/user3 20g >> data/users/user4 10g >> data/share1 100g >> data/share2 200g >> ... >> >> regards, >> ronny >> >> >> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>> I do use cephfs exported via samba. My quota is simply done at the >>> cephfs root though. That works fine with Win10 & Win11 hosts - they >>> see the correct total/used/available space. I haven't tried multiple >>> quotas or setting further down the tree though. >>> >>> Regards, Chris >>> >>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>> hi, hope you can help me. >>>> >>>> i like to export an cephfs via samba an ctdb. >>>> its all working fine via kernel cephfs client. >>>> >>>> i set now quota to the users home dir. >>>> quota also working fine. >>>> >>>> but now, the problem ... on the client machine, i do not see the >>>> quota space. >>>> mostly, i see the complete cluster, or on windows i see nothing >>>> about disk usage. >>>> >>>> it would be important for us and the user, to se the remaining >>>> space. >>>> >>>> samba vfs ceph and ceph_new would not work ... i always got: >>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>> fsname='XXX' No such file or directorymake_connection_snum: >>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file or >>>> directory >>>> >>>> does anybody use cephfs with samba an quota? >>>> running out of ideas, so an help is welcome. >>>> >>>> regards, >>>> ronny >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: cephfs quota samba export
by ronny.lippold 16 Mar '26

16 Mar '26

hi chris and thanks for answer. do you use samba vfs ceph/ceph_new or mount cephfs on samba server and export that? what do you mean, "cephfs root though" i would like to have something like that data/users/user1 10g data/users/user2 5g data/users/user3 20g data/users/user4 10g data/share1 100g data/share2 200g ... regards, ronny Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: > I do use cephfs exported via samba. My quota is simply done at the > cephfs root though. That works fine with Win10 & Win11 hosts - they see > the correct total/used/available space. I haven't tried multiple quotas > or setting further down the tree though. > > Regards, Chris > > On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >> hi, hope you can help me. >> >> i like to export an cephfs via samba an ctdb. >> its all working fine via kernel cephfs client. >> >> i set now quota to the users home dir. >> quota also working fine. >> >> but now, the problem ... on the client machine, i do not see the quota >> space. >> mostly, i see the complete cluster, or on windows i see nothing about >> disk usage. >> >> it would be important for us and the user, to se the remaining space. >> >> samba vfs ceph and ceph_new would not work ... i always got: >> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' fsname='XXX' >> No such file or directorymake_connection_snum: SMB_VFS_CONNECT for >> service 'XXX' at 'PATH' failed: No such file or directory >> >> does anybody use cephfs with samba an quota? >> running out of ideas, so an help is welcome. >> >> regards, >> ronny >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: cephfs quota samba export
by Chris Palmer 16 Mar '26

16 Mar '26

I do use cephfs exported via samba. My quota is simply done at the cephfs root though. That works fine with Win10 & Win11 hosts - they see the correct total/used/available space. I haven't tried multiple quotas or setting further down the tree though. Regards, Chris On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: > hi, hope you can help me. > > i like to export an cephfs via samba an ctdb. > its all working fine via kernel cephfs client. > > i set now quota to the users home dir. > quota also working fine. > > but now, the problem ... on the client machine, i do not see the quota > space. > mostly, i see the complete cluster, or on windows i see nothing about > disk usage. > > it would be important for us and the user, to se the remaining space. > > samba vfs ceph and ceph_new would not work ... i always got: > cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' fsname='XXX' > No such file or directorymake_connection_snum: SMB_VFS_CONNECT for > service 'XXX' at 'PATH' failed: No such file or directory > > does anybody use cephfs with samba an quota? > running out of ideas, so an help is welcome. > > regards, > ronny >

1 0

cephfs quota samba export
by ronny.lippold 16 Mar '26

16 Mar '26

hi, hope you can help me. i like to export an cephfs via samba an ctdb. its all working fine via kernel cephfs client. i set now quota to the users home dir. quota also working fine. but now, the problem ... on the client machine, i do not see the quota space. mostly, i see the complete cluster, or on windows i see nothing about disk usage. it would be important for us and the user, to se the remaining space. samba vfs ceph and ceph_new would not work ... i always got: cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' fsname='XXX' No such file or directorymake_connection_snum: SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file or directory does anybody use cephfs with samba an quota? running out of ideas, so an help is welcome. regards, ronny -- Ronny Lippold System Administrator -- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold(a)spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 --

1 0

Re: Cephadm woes on EVPN loopback network interfaces, detects wrong IPs
by John Mulligan 14 Mar '26

14 Mar '26

On Friday, March 13, 2026 2:43:15 PM Eastern Daylight Time Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > I have a setup here where a server has a single /32 IP from 10.1.0.0/16. It > has a perfectly normal IP address and is reachable from within the local > network thanks to EVPN. cephadm however only detects nonsense: > { > "fe80::/64": { > "data1": [ > "fe80::425b:7fff:fec8:c956" > ], > "loopback0": [ > "fe80::1c54:fbff:fe6a:51e7" > ] > } > } > > How can I set the networks the machine has manually so that it becomes > usable for MON placement? At this time, ceph-mgr does not deploy additional > MONs, for instance, because it claims that the host network does not match > the public network, while I assume the host network is directly read from > the „cephadm list-networks“ output. > Thank you very much in advance Could you share the outputs of: * ip link show * ip addr show * ip route show and * cephadm list-networks (or confirm that the snippet you shared above is the complete output of the command) Thanks!

1 0

Re: Cephadm woes on EVPN loopback network interfaces, detects wrong IPs
by Eugen Block 14 Mar '26

14 Mar '26

Hi, how did you add the node? The usual command is: ceph orch host add <host> <ip> Zitat von Martin Gerhard Loschwitz via ceph-users <ceph-users(a)ceph.io>: > Folks, > > I have a setup here where a server has a single /32 IP from > 10.1.0.0/16. It has a perfectly normal IP address and is reachable > from within the local network thanks to EVPN. cephadm however only > detects nonsense: > > { > "fe80::/64": { > "data1": [ > "fe80::425b:7fff:fec8:c956" > ], > "loopback0": [ > "fe80::1c54:fbff:fe6a:51e7" > ] > } > } > > How can I set the networks the machine has manually so that it > becomes usable for MON placement? At this time, ceph-mgr does not > deploy additional MONs, for instance, because it claims that the > host network does not match the public network, while I assume the > host network is directly read from the „cephadm list-networks“ output. > > Thank you very much in advance > Martin > > -- > Martin Gerhard Loschwitz > Geschäftsführer / CEO, True West IT Services GmbH > Phone: +49 2433 5253130 > Mobile: +49 176 61832178 > Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany > Legal: HRB 21985, Amtsgericht Mönchengladbach > VAT: DE363893844 > > True West IT Services GmbH is compliant with the GDPR regulation on > data protection and privacy in the European Union and the European > Economic Area. You can request the information on how we collect and > process your private data according to the law by contacting the > email sender. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Cephadm woes on EVPN loopback network interfaces, detects wrong IPs
by Martin Gerhard Loschwitz 13 Mar '26

13 Mar '26

Folks, I have a setup here where a server has a single /32 IP from 10.1.0.0/16. It has a perfectly normal IP address and is reachable from within the local network thanks to EVPN. cephadm however only detects nonsense: { "fe80::/64": { "data1": [ "fe80::425b:7fff:fec8:c956" ], "loopback0": [ "fe80::1c54:fbff:fe6a:51e7" ] } } How can I set the networks the machine has manually so that it becomes usable for MON placement? At this time, ceph-mgr does not deploy additional MONs, for instance, because it claims that the host network does not match the public network, while I assume the host network is directly read from the „cephadm list-networks“ output. Thank you very much in advance Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Steve Yates 13 Mar '26

13 Mar '26

And am I reading that right, that after a reboot of an ext4 server TRIM will work again just fine, once? That still seems unrelated to the core issue of the incorrect USED space estimate since that wouldn't apply to Windows, ZFS, etc. Thanks, Steve -----Original Message----- From: Alwin Antreich via ceph-users <ceph-users(a)ceph.io> Sent: Friday, March 13, 2026 8:52 AM To: Steve Yates <steve(a)teamits.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? Hi Steve, On Thu, 12 Mar 2026 at 19:16, Steve Yates <steve(a)teamits.com> wrote: > Hi Alwin, > > > > All VM disks are set as SSD with discard enabled. > > > > “Run guest-trim after a disk move or VM migration” was not checked. I > didn’t realize that was in there! I don’t think that applies though? Per > https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_qemu_agent : > > “With this enabled, Proxmox VE will issue a trim command to the guest > after the following operations that have the potential to write out zeros > to the storage: > > - moving a disk to another storage > - live migrating a VM to another node with local storage” > > > > …and using Ceph the disk doesn’t move if we migrate a VM…? Plus trim is > being run weekly. Maybe I could stop a VM and move its disk to local > storage and back into Ceph again but that would take quite a lot of time > and seems like little to no gain except maybe a lower object count. > That's what I meant as an alternative. > > > That section also has this note on ext4, matching previous discussions: > “There is a caveat with ext4 on Linux, because it uses an in-memory > optimization to avoid issuing duplicate TRIM requests. Since the guest > doesn’t know about the change in the underlying storage, only the first > guest-trim will run as expected. Subsequent ones, until the next reboot, > will only consider parts of the filesystem that changed since then.” > Exactly that. Even if the in-memory bitmap disappears, the underlying misalignment issue will remain and affect any guest. Cheers, Alwin _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdG…)

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Steve Yates 13 Mar '26

13 Mar '26

Hi Alwin, Thanks for the input. Sounds like "stopped" is the safe way, which is what I'd assumed. It would be good if that was clarified in the Ceph docs (https://docs.ceph.com/en/latest/man/8/rbd/#:~:text=sparsify) I found in the Proxmox forum thread that if I create 1+10 GB in two zero files on a small VM, delete, and trim, "rbd du" USED remained at 37/43 GB for that VM, so that doesn't help. Also same issue on Windows VMs, and we have at least one (a pfSense VM) using ZFS, so it's not just EXT4. Re: the serverfault thread, rebooting a VM results in no change in "rbd du" USED. We do regularly reboot VMs for updates, and migrate too when rebooting PVE nodes. I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. -----Original Message----- From: Alwin Antreich via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 11, 2026 6:27 AM To: Steve Yates <steve(a)teamits.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? Hi Steve, I'd advise against using sparsify, especially on any mapped RBD (running VM). We had a support case once where this operation on a live VM caused an OSD lockup, halting the operation for other clients. Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with fstrim (or equivalent) in the guest. rbd du estimates size cheaply by counting the number of objects and multiplying that count by the set object size. Therefore it diverges from the size you see of the guest's filesystem. Some additional points may contribute to the difference: * An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) for discard operations, meaning any discard smaller than this threshold is ignored. * Disk encryption often doesn't clear block for security. * EXT4 uses an in-memory bitmap that does not trim blocks if they haven't been overwritten before. This safeguards against wear on consumer SSDs but has the disadvantage that a storage move will thick-allocate the disk and an fstrim isn't clearing it. The only option is to write a big file and delete it or reboot. See: https://forum.proxmox.com/threads/help-with-trim-on-virtio-scsi-single.1238… I hope this helps. Cheers, Alwin On Tue, 10 Mar 2026 at 14:59, Steve Yates via ceph-users <ceph-users(a)ceph.io> wrote: > Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph > 19/Squid? Or must the VM be shut down first? I tried looking online but > am not finding a direct answer. > > More info: I'm looking into why "rbd du" shows usage inconsistent with > "ceph df" or Proxmox's "usage" displays (Proxmox thread< > https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-…>). > Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, > but doesn't change the amount of space used according to Ceph and PVE. I > am wondering if sparsify can be safely scripted. Or if I should just > ignore "rbd du" USED amounts. > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> TOP 100 Innovator Award <https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdG…)

2 3

Re: Ceph Reef (18.2.4) OSD fails to start with “Permission denied” after Ubuntu 20.04 -> 22.04 upgrade (cephadm)
by Jay See 13 Mar '26

13 Mar '26

Hi Martin, Thanks for the suggestion. I'm not entirely sure what changed during the upgrade. I have more nodes in the same cluster and another cluster with additional nodes where the `ceph` user and group IDs are `64045`, and those are working without any issues. However, on the nodes that were upgraded, the OSDs failed with the permission errors. After changing the `ceph` UID/GID to `167`, everything started working correctly. Once again thanks for your help. ~ Jay. On Fri, Mar 13, 2026 at 1:07 PM Martin Andersen via ceph-users < ceph-users(a)ceph.io> wrote: > > * Jay See via ceph-users <ceph-users(a)ceph.io> on Wed, 11 Mar 2026 at > 09:41: > > > Hi all, > > > > After upgrading a Ceph Reef (18.2.4) node from Ubuntu 20.04 (kernel > 5.4.0) > > to Ubuntu 22.04 (kernel 5.15.0-94), a cephadm-managed OSD fails to start > > with a “Permission denied” error. > > Hi Jay, > > > > > We had a similar issue when moving from 20.04 to 24.04 in a containerized > setup. The issue for us turned out to be that Focal inherited the > uids+gids from > the OS into the containers, whereas Noble did not. Worth checking if it is > the > same on Jammy as well. > > The logs will state that the osd mounts are chowned to ceph:ceph, > sometimes > > > stating the actual numerical uid+gid (167:167). However, this turned out > to be > > > misleading, and one had to manually set the uid+gids on the storage nodes > as > > > these were set to a higher range than what was expected by the containers, > thus > > > causing the “permission denied” error- > > > > > $ usermod -u 167 ceph ; groupmod -g 167 ceph ; reboot > > --Martin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.*

1 0

Re: Ceph Reef (18.2.4) OSD fails to start with “Permission denied” after Ubuntu 20.04 -> 22.04 upgrade (cephadm)
by Martin Andersen 13 Mar '26

13 Mar '26

* Jay See via ceph-users <ceph-users(a)ceph.io> on Wed, 11 Mar 2026 at 09:41: > Hi all, > > After upgrading a Ceph Reef (18.2.4) node from Ubuntu 20.04 (kernel 5.4.0) > to Ubuntu 22.04 (kernel 5.15.0-94), a cephadm-managed OSD fails to start > with a “Permission denied” error. Hi Jay, We had a similar issue when moving from 20.04 to 24.04 in a containerized setup. The issue for us turned out to be that Focal inherited the uids+gids from the OS into the containers, whereas Noble did not. Worth checking if it is the same on Jammy as well. The logs will state that the osd mounts are chowned to ceph:ceph, sometimes stating the actual numerical uid+gid (167:167). However, this turned out to be misleading, and one had to manually set the uid+gids on the storage nodes as these were set to a higher range than what was expected by the containers, thus causing the “permission denied” error- $ usermod -u 167 ceph ; groupmod -g 167 ceph ; reboot --Martin

1 0

Windows driver topics for Monday Steering Committe Re: Ceph Windows driver still has active development?
by Martin Wege 13 Mar '26

13 Mar '26

On Fri, Mar 13, 2026 at 10:23 AM Alexandre Jardon via Dev <dev(a)ceph.io> wrote: > > There is a point on the monday agenda of the Steering Committee to see what will be the ceph posture on this Subject I have a scheduling conflict for that meeting, but I did some research which should be discussed in that call: There are three options to fix this: 1. Fix and improve the existing DOKAN-based driver 2. Implement a new RDBSS-based filesystem driver. This is what SMB, CIFS, RDP, WebDAV, NFSv3, NFSv4.1, BeeFS, VirtualBox SharedFolder drivers do 3. Implement a "raw" Windows driver without the aid of RDBSS. This is what OpenAFS is doing [1] is limited by the features of the Windows DOKAN kernel driver, which excludes Windows ACLs, sparse files Cygwin/MSYS2 support, and a purely synchronous interface. DOKAN and DOKAN-Y also saw almost zero development in the last couple of years. Advantages: It would be a minimum effort to do fixes there. [2] is the most common option for Windows network filesystems, as RDBSS is the kernel utility library for that. But the knowledge of writing a RDBSS driver is mostly limited to Microsoft staff, with the only exceptions being the VirtualBox Shared Folder driver, and the opensource Windows NFSv4.2 driver ms-nfs41-client. Advantages: RDBSS includes support for async IO, ACLs, use of local files as offline cache, sparse file support, Windows EA and Windows named stream support, Cygwin/MSYS2 support. This is a full time job, and requires hiring. [3] would be the most flexible option, because it can do anything a local filesystem driver can do, but likely the most expensive one, as no utility library is involved. All has to be done, and tested, from scratch. Maybe hire one of the OpenAFS guys to do that work. Will also be a full time job. Finally, a solution must be found to get the Windows drivers resulting from [1], [2], or [3] signed with a Secureboot certificate, otherwise the new and improved Ceph Windows driver can only be used with Secureboot disabled, which does not sit well with any enterprise IT setup. Thanks, Martin

1 0

Ceph Windows driver still has active development?
by Sebastian Feld 13 Mar '26

13 Mar '26

Is the Ceph Windows driver still actively being developed, or has the driver been abandoned? Looking at the repro and progress it seems nothing has happened since years: https://github.com/ketor/ceph-dokan Sebi -- Sebastian Feld - IT security consultant

1 0

Re: cephadm: Replacing OSD in mixed setup (block and block.db) is quite messy
by Alex Walender 11 Mar '26

11 Mar '26

Hey Anthony, The NVMEs are Micron 7500 Pro with 7.6TB space (MTFDKCC7T6TGP-1BK1DABYYR). We haven't exactly aimed for multipath disks. Lets say that they were ordered by „accident“ ;) Nevertheless, I was able to pinpoint the issue. It also has something to do with multipath-disks. The issue is described here https://tracker.ceph.com/issues/63862 TL;DR: The inventory command for ceph-volume was not able to list mpath-disks when they were already used as osd. They got falsely filtered out because they got detected as mapper devices with LVs on top. I added a patch-file for this https://tracker.ceph.com/attachments/8006 It appears that this only affects Ceph Reef and older. The code for ceph-volume/util/disk.py has been changed since then. --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens > Am 11.03.2026 um 15:43 schrieb Anthony D'Atri via ceph-users <ceph-users(a)ceph.io>: > >> >> - 36 data disks, registered as mpath-devices >> - 2 NVMEs, which act as block.db for all 36 spinning disks. > > That’s an unusually high ratio. Which model exactly are the NVMe SSDs? And why are you using multipathing with the HDDs? > >> I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: > > Honestly that’s what I often do, especially when applying the orch service to systems with OSDs that were previously deployed under a different scheme. Hybrid OSDs can be messy. > >> service_type: osd >> service_id: delta2024_osd >> service_name: osd.delta2024_osd >> placement: >> label: delta2024 >> spec: >> block_db_size: 397G >> data_devices: >> rotational: 1 >> size: '15T:' >> db_devices: >> rotational: 0 >> encrypted: true >> filter_logic: AND >> objectstore: bluestore >> >> This gives the expected results. >> In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. >> >> Did I find a bug here or is this expected behavior? > > Good question. Bitte enter a ticket at tracker.ceph.com <http://tracker.ceph.com/>. > >> >> >> >> --------------------------- >> M.Sc Alex Walender >> Institut für Bio- und Geowissenschaften >> IBG 5 - Computergestützte Metagenomik / >> de.NBI Cloud Site Bielefeld >> Büro : Universität Bielefeld (UHG), N7-101 >> Tel. : +49-521-106-2907 >> >> Forschungszentrum Jülich GmbH >> 52425 Jülich >> Sitz der Gesellschaft: Jülich >> Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 >> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >> Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

cephfs prone to race conditions?
by Carsten Götze 11 Mar '26

11 Mar '26

Hello! I'm having a problem with cephfs where clients can read past the end of files that have not yet been completely written to the storage. For example if I copy a large file to cephfs and immediately start to copy this large file to another file the second copy will eventually get truncated if the second process is faster than the first one. I have tested both the kernel-driver and the fuse-driver, both with sync and async option and both seem to be prone to the above behavior. Am I missing some mount option here, or is this a bug? With best regards Carsten Götze ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Steve Yates 11 Mar '26

11 Mar '26

If there is any potential danger and no functional gain from sparsify then it doesn't seem like a good idea to run it, and definitely not on a running VM. This cluster was created on 19.2.x. fast-diff and object-map should be on by default. "rbd du" takes a second or two to run. I've been looking at it for a week or so; the USED and OBJECTS change immediately after a sparsify but space used per PVE/"rbd df" doesn't functionally change other than the normal slight variance day to day, or a bit more one day or so each week which I assumed was TRIM. In other words, AFAICT, TRIM operates fine but doesn't change the "rbd du" USED output. Which by extension just increases over time, presumably up to the VM disk size. It will just look weird, eventually having more USED than cluster capacity, if anyone in the distant future runs that command (since PVE shows the actual used space). Currently bluestore_min_alloc_size = 4096 on all OSDs. bluestore_use_optimal_io_size_for_min_alloc_size defaults off, and says "It is a no-op for conventional media" but I can look into it. https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#c… > cool kids skate XFS "I used to be with 'it', but then they changed what 'it' was. Now what I'm with isn't 'it' anymore and what's 'it' seems weird and scary." 👴 -----Original Message----- From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 11, 2026 9:58 AM To: Steve Yates <steve(a)teamits.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? > >> I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. > If all of your clients are the equivalent of Luminous or later, do you have the fast-diff and object-map feature flags enabled by default and for existing volumes? They dramatically increase the speed of `rbd du`. Also, ISTR that there may be an asynchronous GC process where the freed capacity takes some time to show up. Maybe run the du again and see if it’s changed? > Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with > fstrim (or equivalent) in the guest. Agreed. Sparsify is best reserved for volumes that are currently unattached. > An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) This used to default to 64 KiB; in the O/P era these values were split and varied a bit before both settling on 4 KiB. In recent releases they are both 4 KiB to reduce space amp, e.g. for tiny RGW objects. Note that despite some references on the net, you cannot effectively retrofit this value for an existing OSD. The process will report a runtime change, but it’s actually baked into the OSD and cannot be changed in practice without redeploying. From … Quincy I think it was, the baked-in value is reported by `ceph osd metadata`. Some outdated references suggest setting the value based on workload, I suggest not doing that unless you are using coarse-IU SSDs e.g. P4326, P5316, P5336, 6550, et al. I suggest setting bluestore_use_optimal_io_size_for_min_alloc_size to true before creating OSDs. I have never seen a case where it does the wrong thing. > EXT4 uses > All the cool kids skate XFS, warts and all ;) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdG…)

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Anthony D'Atri 11 Mar '26

11 Mar '26

> >> I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. > If all of your clients are the equivalent of Luminous or later, do you have the fast-diff and object-map feature flags enabled by default and for existing volumes? They dramatically increase the speed of `rbd du`. Also, ISTR that there may be an asynchronous GC process where the freed capacity takes some time to show up. Maybe run the du again and see if it’s changed? > Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with > fstrim (or equivalent) in the guest. Agreed. Sparsify is best reserved for volumes that are currently unattached. > An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) This used to default to 64 KiB; in the O/P era these values were split and varied a bit before both settling on 4 KiB. In recent releases they are both 4 KiB to reduce space amp, e.g. for tiny RGW objects. Note that despite some references on the net, you cannot effectively retrofit this value for an existing OSD. The process will report a runtime change, but it’s actually baked into the OSD and cannot be changed in practice without redeploying. From … Quincy I think it was, the baked-in value is reported by `ceph osd metadata`. Some outdated references suggest setting the value based on workload, I suggest not doing that unless you are using coarse-IU SSDs e.g. P4326, P5316, P5336, 6550, et al. I suggest setting bluestore_use_optimal_io_size_for_min_alloc_size to true before creating OSDs. I have never seen a case where it does the wrong thing. > EXT4 uses > All the cool kids skate XFS, warts and all ;)

1 0

Re: cephadm: Replacing OSD in mixed setup (block and block.db) is quite messy
by Anthony D'Atri 11 Mar '26

11 Mar '26

> > - 36 data disks, registered as mpath-devices > - 2 NVMEs, which act as block.db for all 36 spinning disks. That’s an unusually high ratio. Which model exactly are the NVMe SSDs? And why are you using multipathing with the HDDs? > I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: Honestly that’s what I often do, especially when applying the orch service to systems with OSDs that were previously deployed under a different scheme. Hybrid OSDs can be messy. > service_type: osd > service_id: delta2024_osd > service_name: osd.delta2024_osd > placement: > label: delta2024 > spec: > block_db_size: 397G > data_devices: > rotational: 1 > size: '15T:' > db_devices: > rotational: 0 > encrypted: true > filter_logic: AND > objectstore: bluestore > > This gives the expected results. > In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. > > Did I find a bug here or is this expected behavior? Good question. Bitte enter a ticket at tracker.ceph.com <http://tracker.ceph.com/>. > > > > --------------------------- > M.Sc Alex Walender > Institut für Bio- und Geowissenschaften > IBG 5 - Computergestützte Metagenomik / > de.NBI Cloud Site Bielefeld > Büro : Universität Bielefeld (UHG), N7-101 > Tel. : +49-521-106-2907 > > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle 20.2.1 QE validation status
by Ilya Dryomov 11 Mar '26

11 Mar '26

On Mon, Mar 9, 2026 at 11:38 PM Ilya Dryomov via ceph-users <ceph-users(a)ceph.io> wrote: > > On Mon, Mar 9, 2026 at 7:50 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/74838#note-4 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > Hi Yuri, > > Could you please do another rerun for rbd? rbd approved based on https://pulpito.ceph.com/yuriw-2026-03-10_15:51:59-rbd-tentacle-release-dis… Thanks, Ilya

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Alwin Antreich 11 Mar '26

11 Mar '26

Hi Steve, I'd advise against using sparsify, especially on any mapped RBD (running VM). We had a support case once where this operation on a live VM caused an OSD lockup, halting the operation for other clients. Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with fstrim (or equivalent) in the guest. rbd du estimates size cheaply by counting the number of objects and multiplying that count by the set object size. Therefore it diverges from the size you see of the guest's filesystem. Some additional points may contribute to the difference: * An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) for discard operations, meaning any discard smaller than this threshold is ignored. * Disk encryption often doesn't clear block for security. * EXT4 uses an in-memory bitmap that does not trim blocks if they haven't been overwritten before. This safeguards against wear on consumer SSDs but has the disadvantage that a storage move will thick-allocate the disk and an fstrim isn't clearing it. The only option is to write a big file and delete it or reboot. See: https://forum.proxmox.com/threads/help-with-trim-on-virtio-scsi-single.1238… I hope this helps. Cheers, Alwin On Tue, 10 Mar 2026 at 14:59, Steve Yates via ceph-users <ceph-users(a)ceph.io> wrote: > Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph > 19/Squid? Or must the VM be shut down first? I tried looking online but > am not finding a direct answer. > > More info: I'm looking into why "rbd du" shows usage inconsistent with > "ceph df" or Proxmox's "usage" displays (Proxmox thread< > https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-…>). > Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, > but doesn't change the amount of space used according to Ceph and PVE. I > am wondering if sparsify can be safely scripted. Or if I should just > ignore "rbd du" USED amounts. > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> TOP 100 Innovator Award <https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte

1 0

Ceph Reef (18.2.4) OSD fails to start with “Permission denied” after Ubuntu 20.04 -> 22.04 upgrade (cephadm)
by Jay See 11 Mar '26

11 Mar '26

Hi all, After upgrading a Ceph Reef (18.2.4) node from Ubuntu 20.04 (kernel 5.4.0) to Ubuntu 22.04 (kernel 5.15.0-94), a cephadm-managed OSD fails to start with a “Permission denied” error. I’ve checked for related bugs, including *tracker.ceph.com/issues/66989#note-8 <http://tracker.ceph.com/issues/66989#note-8>* , and even tried adding "*--security-opt seccomp=unconfined*", but the issue persists. There are more than one OSD, just sharing the details of one of the OSD here. Syslogs from a failed OSD start are available here: *https://paste.ubuntu.com/p/8n52TJdcSt/ <https://paste.ubuntu.com/p/8n52TJdcSt/>. * ~$ ceph orch ps --hostname h006 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.9 h006 error 22h ago 4y - 4096M <unknown> <unknown> Can anyone point me to a solution, or to previous discussions or bugs addressing this problem? Thanks in advance, Jay.

1 0

cephadm: Replacing OSD in mixed setup (block and block.db) is quite messy
by Alex Walender 11 Mar '26

11 Mar '26

Hey friends, Our production system is still running ceph with manual Debian packages. Since I’d like to migrate to a deployment based on cephadm, I experimented the migration and usage of day 1 ops on our development environment. Both systems run with v18.2.7. The migration of the dev-system went well. The only headache I have is with replacing OSDs (simulating the replacement of a faulty disk). The osd-node has the following setup: - 36 data disks, registered as mpath-devices - 2 NVMEs, which act as block.db for all 36 spinning disks. Therefore each NVME holds 18 LVs. For the deployment I used a very simple file: ``` service_type: osd service_id: delta2024_osd placement: label: delta2024 spec: data_devices: size: '15T:' rotational: 1 db_devices: size: '6T:' rotational: 0 objectstore: bluestore encrypted: true ``` This deploys all osds as expected. On cephadm-logs, I see the „ceph-volume lvm batch“ command, including all 36 disks as arguments. Now I want to simulate the replacement of a failing disk: - Set the deployment to unmanaged via „ceph orch set-unmanaged …“ - Zap one disk with „ceph orch osd rm 56 —zap --replace“ - „vgs“ show, that one of the NVMEs has space again. - „ceph orch device ls —wide —refresh“ shows the zapped disk as available again - Set the deployment to managed again „ceph orch set-managed …“ - I expect, that the OSD gets re-deployed again, but nothing happens. This time, cephadm sends a shorter command to my osd-node: „ceph-volume lvm batch --auto /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 --no-systemd —yes“ This exits with the message, that none of the passed fast-devices are available. After digging in the code for ceph-volume, I may found an issue: Since only one disk is included in the batch call (/dev/mapper/mpathz), ceph-volume tries to calculate a whole NVME disk for its block.db. This, of course, doesn’t work. It has no clue on how to properly slice a LV for the new block.db on an existing NVME drive given the passed arguments. The check that fails is this snippet (ceph_volume/devices/lvm/batch.py): while abs_size <= free_size and len(ret) < new_osds and occupied_slots < fast_slots_per_device: free_size -= abs_size.b occupied_slots += 1 ret.append((dev.path, relative_size, abs_size, requested_slots)) When using an empty node, occupied_slots = 0, and fast_slots_per_device = 18. Therefore the deployment works. When only replacing one OSD, occupied_slots = 17 (because one LV got zapped) and fast_slots_per_device = 1. Therefore the last conditional check is false, the loop is never entered and no OSD is created as result. I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: ``` service_type: osd service_id: delta2024_osd service_name: osd.delta2024_osd placement: label: delta2024 spec: block_db_size: 397G data_devices: rotational: 1 size: '15T:' db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore ``` This gives the expected results. In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. Did I find a bug here or is this expected behavior? --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

1 0

Re: Broken dashboard after upgrade to 20.2.0
by Ankush Behl 11 Mar '26

11 Mar '26

I assume this is happening on a cluster that is running on debian. Tracker: https://tracker.ceph.com/issues/74199 PR for Fix: https://github.com/ceph/ceph/pull/67133 We have raised a backport PR in the Tentacle branch to fix it. https://github.com/ceph/ceph/pull/67510 On Wed, Feb 25, 2026 at 12:25 PM Nicola Mori via ceph-users < ceph-users(a)ceph.io> wrote: > Any help? > > On 18/02/26 8:17 AM, Nicola Mori wrote: > > I found that the error is reproducible from Ceph CLI: > > > > # ceph dashboard get-prometheus-api-host > > Error EIO: Module 'dashboard' has experienced an error and cannot handle > > commands: No module named 'smb' > > > > Any hint? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

An appeal for community help: mailing lists
by Patrick Donnelly 10 Mar '26

10 Mar '26

Hello, As you may have noticed, the ceph mailing lists have struggled with reliable delivery of mail postings. Additionally, emails sent to the lists are rewritten to comply with DMARC authenticity checks. This means you will get duplicates frequently. Unfortunately the Ceph project lacks modern expertise and time to make mailing lists work with the new rules pioneered by the email monopolists. We really could use some help from knowledgeable folks who would be willing to manage or advise on software (we are currently using GNU Mailman) and configurations that will work. Ideally, like LKML, the mailing list should be a dumb reflector as much as possible (no mail rewrites!). If you would be willing to contribute your time towards this, please get in touch with me privately off-list. Thanks, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: reef 18.2.8 QE validation status
by Patrick Donnelly 10 Mar '26

10 Mar '26

The `fs` and `powercycle` suites will need rerun with [1]. I believe you're already doing that but I'm documenting that for the list. I'm going to look into the upgrade issues today [2]. [1] https://github.com/ceph/ceph/pull/67495 [2] https://tracker.ceph.com/issues/75034 On Mon, Feb 23, 2026 at 4:29 PM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > Patrick, > > The only outstanding issues are the FS approval and upgrade issues, > reported by Laura. > > What do you suggest we do next here? > > On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > > > orch approved. Run looks ugly due to us no having a proper ignorelist for > > the orch suite on reef, but the actual results are okay > > > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Rerun scheduled for krbd > >> > >> Still waiting for replies from dev leads > >> > >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: > >> > >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> > >> > wrote: > >> > > > >> > > Note: this is an attempt to test and publish post EOL reef point > >> release > >> > > > >> > > Details of this release are summarized here: > >> > > > >> > > https://tracker.ceph.com/issues/73906#note-1 > >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > >> > > Release Notes - TBD > >> > > LRC upgrade - N/A > >> > > > >> > > distro => jammy focal centos9 windows bookworm > >> > > > >> > > Seeking approvals/reviews for: > >> > > > >> > > smoke - Laura approved? > >> > > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > >> approved? > >> > > (rados is also running on build 2 == build 1 + > >> > > https://github.com/ceph/ceph/pull/66056 > >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2…> > >> per Igor's request) > >> > > > >> > > rgw - Adam E approved? > >> > > > >> > > fs - Venky approved? > >> > > > >> > > orch - Adam King approved? > >> > > > >> > > rbd - Ilya approved? > >> > > krbd - Ilya approved? > >> > > >> > rbd approved, krbd needs a rerun as all jobs failed while > >> > going through cephlab.yml playbook (not related to RBD): > >> > > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, > >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > >> > codepage or helper program, or other error.'] > >> > > >> > Thanks, > >> > > >> > Ilya > >> > > >> > > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Can rbd sparsify be run on a running VM's disk?
by Steve Yates 10 Mar '26

10 Mar '26

Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph 19/Squid? Or must the VM be shut down first? I tried looking online but am not finding a direct answer. More info: I'm looking into why "rbd du" shows usage inconsistent with "ceph df" or Proxmox's "usage" displays (Proxmox thread<https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-…>). Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, but doesn't change the amount of space used according to Ceph and PVE. I am wondering if sparsify can be safely scripted. Or if I should just ignore "rbd du" USED amounts. Thanks.

1 0

Re: tentacle 20.2.1 QE validation status
by Afreen 10 Mar '26

10 Mar '26

Dashboard approved. There is a known issue which is a flake being fixed here https://tracker.ceph.com/issues/70669 On Tue, Mar 10, 2026 at 12:17 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya (still running) > nvmeof: Aviv, Vallari > > upgrade/squid-x: Laura > upgrade/reef-x: Laura > upgrade/telemetry-upgrade: Laura > > upgrade/tentacle-p2p: N/A > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: tentacle 20.2.1 QE validation status
by Ilya Dryomov 09 Mar '26

09 Mar '26

On Mon, Mar 9, 2026 at 7:50 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya Hi Yuri, Could you please do another rerun for rbd? > krbd: Ilya (still running) krbd approved. Thanks, Ilya

1 0

CSC Meeting Notes: 2026-03-09
by Adam Emerson 09 Mar '26

09 Mar '26

Ceph Steering Committee Meeting 9 March 2026 * [Ernesto/AnthonyM]: Github Plan migration to Enterprise Account. - Upgrading Github to enterprise is generally pretty straightforward - The main issue is permissions with github actions - Benjamin Sternthal from the LF can help audit and get things ready. - The main issue is default permissions and needing to update actions to specify them. - [Ilya] We don't have all that many actions, so some temporary breakage probably wouldn't be the end of the world. - [Benjamin Sternthal] Probably the best thing to do is pick a time and work back from there. - The banner from Github suggests it's going to happen automatically eventually, and it's better if we do it on our timeline. - LF Requested the upgrade about a week ago, it usually comes in two weeks, so it should be soon. - #github-enterprise-upgrade is open for anyone wanting to be involved. * [Anthony Middleton] CSC Election https://docs.google.com/document/d/12zmuG6qVWU33dlcR4aXkCG2z4MdAGT8A3F36ofn… - Continuing from previous conversations, this is a planning document for how the election could go. - Lots of questions about voting. - There seems to be a consensus about having up to three co-chairs. If we want to have a vote about that, Anthony's happy and Benjamin Sternthal says there's no LF objection. - This election is happening now because we didn't have an election on 2025, so there will be another election later this year as well. - We could have an election about leadership now, and handle other matters around October/November. - Do we need to move forward with the leadership? Or is there more that needs done soon? - [Greg] We need to clean up our rolls. Anthony has a list of who voted in the 2024/2025 elections. - We can make a list of people who didn't vote in the last two elections and vote about removing them. - [Benjamin Sternthal] Now would be a good time to think about how to run things, conduct votes, etc. - [Dan van der Ster] We could do concrete procedural changes in pull requests, put our procedural documents on github, so we can iterate and vote on them. That's how we did it in the past with our governance rules. - [Matt V] ARIN/RIPE revokes voting for orgs that haven't voted in two years, which sounds similar. - Mark Nelson and Anthony will work on a draft. We've talked about auto-expiration, an annual event where people have to affirm their voting status (but there were concerns the note might be missed), or we vote every year on pruning our rolls. - [Ilya] We have to vote once a year to rotate council members. We could have whoever's organizing that vote propose removal of inactive members (based on some rules) as part of the same ballot. If we have to vote every year anyway, then doing it then will go much more smoothly. - [Dan van der Ster] This is how we did it last time, we just didn't have a specific procedure for determining inactive members. - [Anthony Middleton] Who will represent the CSC to the quarterly meeting of the Ceph Governing Board at the LF? That could be one of the leadership people or any member of the CSC. - [Dan van der Ster] The CEC can decide among themselves who will be the voting representative/single point of contact. - [Benjamin Sternthal] There's a Voting Representative and this organization decides who it is. The whole CSC is invited to the quarterly meetings with the Governing Board. - [Anthony Middleton] So the vote will be about leadership and the removal process for inactive members. Then there will be another election in October or November. * [Mark] Do we have a policy on which QA runs should pass before PRs are merged? continue discussion from last meeting. - Mark remembered Ilya saying there was a policy somewhere. - Ilya qualified that he was saying the general consensus, not a policy. Components scheduling test runs, with merges happening only after test runs are finished and reviewed. - The point was that you can do your own run, but it should be a coordinated process where the relevant component team reviews the results, since not everyone is up to speed on all of the ongoing issues. - [Mark] We know what the expectaiton from the leads is, but don't have anything written down. - [Greg] Someone should just write something up and put up a PR - [Laura Flores] It should probably be component-by-component. Component leads should write down their own process. - Adam E will get it started and the component leads can fill it in. * [Radoslaw] Conclusion on "Should the `reef` branch be opened for merging" question. - Moving to next week since Patrick isn't here * [Anthony Middleton] CSC and Ceph Governing Board Q1 meeting, Monday March 30th 1 pm EDT - [Greg] The agenda looks very long and that is not a one hour meeting - Anthony Middleton will edit the agenda, moderate the call, etc. - This meeting will be led by the governing board this quarter, next quarter the CSC will be. * [Patrick] Shall we archive ceph/ceph-deploy and other vestiges? - Bumped to next week since Patrick isn't here. * [Yaarit] Rocky 10 update - We are working on merging the patches for having Rock 10 in main and the Python 3.12 changes. - Shortly after, we would like to change the base image to be Rocky 10 instead of CentOS 9. - Will have a separate discussion with David G, but wants to ask all the component leads to review and fix the test branches. - We saw an unreasonable amount of failures last week, and we're currently looking into that. - Once everything looks green, we hope to be able to merge the changes. - There are pending issues for CephFS and RBD as well.

1 0

Re: RAW USE in 'df' - no pools
by Janne Johansson 07 Mar '26

07 Mar '26

I think it counts overhead "lost" to preallocated space and databases which occur even before a single pool is in place. Den lör 7 mars 2026 kl 09:28 skrev listy via ceph-users <ceph-users(a)ceph.io>: > > Hi guys. > > What is 'RAW USE' in _df_ while there is no pool using given > _osd_? > Like so: > > -> $ ceph osd df tree | egrep '(osd\.(3|2|4)\ |ID)' > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA > OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME > 2 MONERO 0.39059 1.00000 400 GiB 660 MiB 14 MiB > 31 KiB 646 MiB 399 GiB 0.16 0.04 0 up > osd.2 > 3 MONERO 0.39059 1.00000 400 GiB 1.5 GiB 15 MiB > 467 KiB 1.4 GiB 399 GiB 0.36 0.08 1 up > osd.3 > 4 MONERO 0.39059 1.00000 400 GiB 567 MiB 15 MiB > 50 KiB 553 MiB 399 GiB 0.14 0.03 1 up > osd.4 > > I have crush rule which is mean to - at least how I get it - > separate osds: cluster is 3x nodes with 150GB + 400GB drives > on each node - in case this matter, separation is suppose to > be between different size drives. > Should _raw use_ & _data_ not be always _zero_? > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- May the most significant bit of your life be positive.

1 0

Set Up Ceph certmgr CA as SubCA of an internal RootCA
by Kleinknecht_ch＠web.de 07 Mar '26

07 Mar '26

Hello cephusers Does anyone have experience setting up the Ceph certmgr CA as a SubCA of an internal RootCA? I haven't found anything in the documentation yet. Greetings Christoph

1 0

Re: RAW USE in 'df' - no pools
by Eugen Block 07 Mar '26

07 Mar '26

Hi, each OSD has a DB/WAL device which is on the main data device if not separated on faster drives. And since there are two PGs, there is at least one pool, I assume the .mgr pool which is created by default. Regards, Eugen Zitat von listy via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > What is 'RAW USE' in _df_ while there is no pool using given _osd_? > Like so: > > -> $ ceph osd df tree | egrep '(osd\.(3|2|4)\ |ID)' > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META AVAIL %USE VAR PGS STATUS TYPE NAME > 2 MONERO 0.39059 1.00000 400 GiB 660 MiB 14 MiB 31 KiB > 646 MiB 399 GiB 0.16 0.04 0 up osd.2 > 3 MONERO 0.39059 1.00000 400 GiB 1.5 GiB 15 MiB 467 KiB > 1.4 GiB 399 GiB 0.36 0.08 1 up osd.3 > 4 MONERO 0.39059 1.00000 400 GiB 567 MiB 15 MiB 50 KiB > 553 MiB 399 GiB 0.14 0.03 1 up osd.4 > > I have crush rule which is mean to - at least how I get it - > separate osds: cluster is 3x nodes with 150GB + 400GB drives on each > node - in case this matter, separation is suppose to be between > different size drives. > Should _raw use_ & _data_ not be always _zero_? > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

RAW USE in 'df' - no pools
by listy 07 Mar '26

07 Mar '26

Hi guys. What is 'RAW USE' in _df_ while there is no pool using given _osd_? Like so: -> $ ceph osd df tree | egrep '(osd\.(3|2|4)\ |ID)' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 2 MONERO 0.39059 1.00000 400 GiB 660 MiB 14 MiB 31 KiB 646 MiB 399 GiB 0.16 0.04 0 up osd.2 3 MONERO 0.39059 1.00000 400 GiB 1.5 GiB 15 MiB 467 KiB 1.4 GiB 399 GiB 0.36 0.08 1 up osd.3 4 MONERO 0.39059 1.00000 400 GiB 567 MiB 15 MiB 50 KiB 553 MiB 399 GiB 0.14 0.03 1 up osd.4 I have crush rule which is mean to - at least how I get it - separate osds: cluster is 3x nodes with 150GB + 400GB drives on each node - in case this matter, separation is suppose to be between different size drives. Should _raw use_ & _data_ not be always _zero_? many thanks, L.

1 0

Re: rbd export/export-diff of group snaps
by Ilya Dryomov 06 Mar '26

06 Mar '26

On Fri, Mar 6, 2026 at 8:04 AM Sascha Lucas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Ilya, > > On Thu, 5 Mar 2026, Ilya Dryomov via ceph-users wrote: > > > To see the image-level snapshots that comprise a group snapshot taken > > with "rbd group snap create", run "rbd snap ls --all". > > > > There is no support for cloning from a group snapshot directly but > > cloning from the corresponding image-level snapshots is possible with > > "rbd clone <image name> --snap-id <snap id>" syntax (instead of the > > usual "rbd clone <image name>@<snap name>"). As a workaround one can > > do that for each member image and then create a new group out of the > > resulting clone images. The new "clone" group wouldn't get linked to > > the original "parent" group like clone images get linked to their > > parent images though. > > Thanks for this valuable information. I was able to clone each individual > image from the group snapshot. Also found this IBM documentation[1]. > > > Unfortunately there is no support for exporting a group snapshot > > directly (you already saw the unfinished PR) and even being able to > > export the corresponding image-level snapshots individually is pending > > on https://tracker.ceph.com/issues/72410. As a workaround one can > > clone from an image-level snapshot and then export the clone image. > > Also confirming that exporting the cloned images works well. But with this > workaround there seems no chance for export-diff between two group > snapshots? Correct -- https://tracker.ceph.com/issues/72410 is needed for that. Thanks, Ilya

1 0

Question about caching with Ceph
by Ewen Chan 06 Mar '26

06 Mar '26

Hi there. I am looking to potentially evaluate migrating my storage systems from ZFS to Ceph and my question is, is there something like ZFS' adaptive replacement cache (ARC) on Ceph? I am asking because I am looking to potentially deploy four nodes with six HGST 6 TB SATA HDDs in each node, where each HDD will become a Ceph OSD. My concern is that without something like ZFS' ARC, if it has to read the data from disk every single time, that it can be really slow, especially for repeat reads. I found this old thread talking a little bit about this: https://ceph-users.ceph.narkive.com/vPnNl3zj/cephfs-and-caching#, but as noted, it is a rather old thread by now, so I am wondering what is the state of caching with Ceph? I understand that cache tiering has been deprecated due to the lack of a maintainer. It is also my understanding (via the above mentioned older thread) that ZFS ARC works a little differently than the "general" linux cache. Your help in educating me how this works in 2026, is greatly appreciated. Thank you. Sincerely ~alpha

1 0

Re: rbd export/export-diff of group snaps
by Sascha Lucas 06 Mar '26

06 Mar '26

Hi Ilya, On Thu, 5 Mar 2026, Ilya Dryomov via ceph-users wrote: > To see the image-level snapshots that comprise a group snapshot taken > with "rbd group snap create", run "rbd snap ls --all". > > There is no support for cloning from a group snapshot directly but > cloning from the corresponding image-level snapshots is possible with > "rbd clone <image name> --snap-id <snap id>" syntax (instead of the > usual "rbd clone <image name>@<snap name>"). As a workaround one can > do that for each member image and then create a new group out of the > resulting clone images. The new "clone" group wouldn't get linked to > the original "parent" group like clone images get linked to their > parent images though. Thanks for this valuable information. I was able to clone each individual image from the group snapshot. Also found this IBM documentation[1]. > Unfortunately there is no support for exporting a group snapshot > directly (you already saw the unfinished PR) and even being able to > export the corresponding image-level snapshots individually is pending > on https://tracker.ceph.com/issues/72410. As a workaround one can > clone from an image-level snapshot and then export the clone image. Also confirming that exporting the cloned images works well. But with this workaround there seems no chance for export-diff between two group snapshots? > See my reply to Matthias -- involving qemu-guest-agent to freeze the > filesystems before taking the snapshot is actually a superior option, > so I wouldn't refer to it as a workaround. Yes, immediately after sending, I realised that "workaround" is not the right wording. Of cause is fs-freezing the next right step into more consistent snapshots. Thanks, Sascha. [1] https://www.ibm.com/docs/en/storage-ceph/9.9.0?topic=snapshot-cloning-block…

1 0

Re: rbd export/export-diff of group snaps
by Ilya Dryomov 05 Mar '26

05 Mar '26

On Thu, Mar 5, 2026 at 6:14 PM Sascha Lucas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Matthias, > > On Thu, 5 Mar 2026, Matthias Riße via ceph-users wrote: > > > I ran into this issue too and was quite puzzled. There is this open PR which > > suggests that group snapshot exports are unsupported as of now: > > https://github.com/ceph/ceph/pull/40808 > > Thanks for your fast reply. I was afraid of that the group feature is not > yet much usable for: no export, no clone, no mirror. Just snap/rollback > works. The IBM version supports group mirror[1]. But I don't know if this > feature will stay an exclusive enterprise one? > > The mentioned PR is old and outdated. Seems not a prominent feature ... Hi Sascha, To see the image-level snapshots that comprise a group snapshot taken with "rbd group snap create", run "rbd snap ls --all". There is no support for cloning from a group snapshot directly but cloning from the corresponding image-level snapshots is possible with "rbd clone <image name> --snap-id <snap id>" syntax (instead of the usual "rbd clone <image name>@<snap name>"). As a workaround one can do that for each member image and then create a new group out of the resulting clone images. The new "clone" group wouldn't get linked to the original "parent" group like clone images get linked to their parent images though. Unfortunately there is no support for exporting a group snapshot directly (you already saw the unfinished PR) and even being able to export the corresponding image-level snapshots individually is pending on https://tracker.ceph.com/issues/72410. As a workaround one can clone from an image-level snapshot and then export the clone image. > > > The best thing I could figure out to get consistent snapshots with multiple > > rbd's is to fsfreeze my filesystems (using libvirt, so domfs{freeze,thaw} in > > my case) and make individual snapshots of the rbd images. > > Yes, I already considered this "workaround", but I'm not using libvirt, so > have to talk to qemu-guest-agent my self. See my reply to Matthias -- involving qemu-guest-agent to freeze the filesystems before taking the snapshot is actually a superior option, so I wouldn't refer to it as a workaround. Thanks, Ilya

1 0

Re: rbd export/export-diff of group snaps
by Ilya Dryomov 05 Mar '26

05 Mar '26

On Thu, Mar 5, 2026 at 4:43 PM Matthias Riße via ceph-users <ceph-users(a)ceph.io> wrote: > > Hey Sascha, > > I ran into this issue too and was quite puzzled. There is this open PR > which suggests that group snapshot exports are unsupported as of now: > https://github.com/ceph/ceph/pull/40808 > > The best thing I could figure out to get consistent snapshots with > multiple rbd's is to fsfreeze my filesystems (using libvirt, so > domfs{freeze,thaw} in my case) and make individual snapshots of the rbd > images. Hi Matthias, I just want to note that freezing the filesystem is indeed a very good thing because in the end (whether you then take snapshots individually or with "rbd group snap create") you get a set of image snapshots that are filesystem-consistent. "rbd group snap create" in isolation would produce only a crash-consistent set of image snapshots. Thanks, Ilya

1 0