- ceph-users - lists.ceph.io

Re: Restricting subuser to its bucket
by Eugen Block 30 Mar '26

30 Mar '26

Hi, I would suggest to run the aws cp command with --debug to see more details, the message "argument of type 'NoneType' is not iterable" is not really explanatory. ;-) Regards, Eugen "Rok Jaklič via ceph-users" ceph-users(a)ceph.io – 18. März 2026 10:53 > Hi, > > I am trying to restrict subuser to its bucket (so he/she can work only on a > bucket which has permission) with disabling ACLs and setting policy on a > bucket: > > Using ceph version 19.2.2 > > bucket1-policy.json contains > { > "Version": "2012-10-17", > "Statement": [ > { > "Effect": "Allow", > "Principal": { > "AWS": "arn:aws:iam::tenant1:user/parent1:user1" > }, > "Action": "s3:*", > "Resource": [ > "arn:aws:s3:::bucket1", > "arn:aws:s3:::bucket1/*" > ] > } > ] > } > > --- > > echo "lala" > test.txt > > export RGW_ENDPOINT="" > > radosgw-admin user create --tenant=tenant1 --uid=parent1 --key-type=s3 > --display-name="tenant1:parent1" > for sub in user1 user2 user3; do > radosgw-admin subuser create --tenant=tenant1 --uid=parent1 --subuser=$sub > --key-type=s3 --access=full; > done > > export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user1").access_key') > export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user1").secret_key') > > aws s3api create-bucket --bucket bucket1 --object-ownership > BucketOwnerEnforced --endpoint-url $RGW_ENDPOINT > aws s3api put-bucket-policy --bucket bucket1 --policy > file://bucket1-policy.json --endpoint-url $RGW_ENDPOINT > aws s3api put-public-access-block --bucket bucket1 > --public-access-block-configuration > BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true > --endpoint-url $RGW_ENDPOINT > > aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url $RGW_ENDPOINT > aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT > > # verify > export AWS_ACCESS_KEY_ID=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user2").access_key') > export AWS_SECRET_ACCESS_KEY=$(radosgw-admin user info --uid="parent1" > --tenant="tenant1" | jq .keys | jq -r '.[] | > select(.user=="tenant1$parent1:user2").secret_key') > > aws s3 ls s3://bucket1 --endpoint-url $RGW_ENDPOINT > > --- > > Script stops at aws s3 cp test.txt s3://bucket1/test.txt --endpoint-url > $RGW_ENDPOINT with an error like: > upload failed: ./test.txt to s3://bucket1/test.txt argument of type > 'NoneType' is not iterable > > If I comment the line with aws s3api put-public-access-block... I can > successfully upload some file, but I can also see the contents of bucket1 > with user2, which is not what I want. > > Any ideas how would I restrict the subuser to its bucket only? > > I was actually able to restrict subuser to its bucket by explicitly denying > access to bucket with following policy: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/AFNERFMNSC… > > But it doesn't feel the right way. Wouldn't be better to "restrict" > everything for everyone but its user in the first place and then allow > others to access? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Crash on newly created NVMe+HDD OSD
by Jan Kasprzak 30 Mar '26

30 Mar '26

Hello, Ceph users, I wanted to replace a HDD with bad blocks in an OSD which has block.db on NVMe. Removing the OSD (ceph-volume lvm zap, etc.) went well, no VG/LV remained for data or metadata. But after replacing the HDD, preparing a new HDD+NVMe OSD, and starting it with "ceph-volume lvm activate --all", the OSD daemon crashed with terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input Retrying the proces once more I noticed that ceph-volume created a VG/PV/LV on the HDD (/dev/sda), but did not create a VG/PV/LV on /dev/nvme1n1p4. Zapping the result once more and creating a HDD-only OSD helped, the OSD daemon started and joined the cluster correctly. I then stopped it once more, and tried to create a db volume myself, and migrate db to it: systemctl stop ceph-osd@$OSD_ID NEW_DB_VG=ceph-`uuidgen` NEW_DB_LV=osd-db-`uuidgen` vgcreate $NEW_DB_VG /dev/nvme1n1p4 lvcreate -a y -l 100%FREE -n $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm new-db --osd-id $OSD_ID --osd-fsid $OSD_FSID --target $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm migrate --osd-id $OSD_ID --osd-fsid $OSD_FSID --from data --target $NEW_DB_VG/$NEW_DB_LV systemctl start ceph-osd@$OSD_ID ... and it started correctly and did not crash. I used the same commands previously on Ceph 19 when replacing HDDs, and it worked without problem. This is the first time I replaced a HDD after upgrading to Ceph 20.2.0, so it might be a regression in ceph-volume of 20.2.0. Below is the output of "ceph-volume lvm prepare", and the crash report from journal. Did I do something incorrect here? Thanks! -Yenya # ceph-volume lvm prepare --bluestore --block.db /dev/nvme1n1p4 --data /dev/sda Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: vgcreate --force --yes ceph-8805a195-57fd-4054-8282-a307d15955a5 /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-8805a195-57fd-4054-8282-a307d15955a5" successfully created Running command: lvcreate --yes -l 2861055 -n osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 ceph-8805a195-57fd-4054-8282-a307d15955a5 stdout: Logical volume "osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963" created. Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-25 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 Running command: /usr/bin/ln -s /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 /var/lib/ceph/osd/ceph-25/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-25/activate.monmap stderr: got monmap epoch 39 --> Creating keyring file for osd.25 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/keyring Running command: /usr/bin/chown -R ceph:ceph /dev/nvme1n1p4 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 25 --monmap /var/lib/ceph/osd/ceph-25/activate.monmap --keyfile - --bluestore-block-db-path /dev/nvme1n1p4 --osd-data /var/lib/ceph/osd/ceph-25/ --osd-uuid db38dbfb-e161-4680-93e3-ba4abaa36963 --setuser ceph --setgroup ceph stderr: 2026-03-30T07:59:06.150+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25//block) No valid bdev label found stderr: 2026-03-30T07:59:06.175+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/sda Mar 30 07:52:31 osdhost systemd[1]: Starting Ceph object storage daemon osd.25... Mar 30 07:52:31 osdhost systemd[1]: Started Ceph object storage daemon osd.25. Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' Mar 30 07:52:31 osdhost ceph-osd[1140285]: what(): static void bluefs_fnode_t::_denc_finish(ceph::buffer::v15_2_0::ptr::const_iterator&, char**, uint32_t*): Malformed input [buffer:3] Mar 30 07:52:31 osdhost ceph-osd[1140285]: *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Main process exited, code=dumped, status=6/ABRT Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Failed with result 'core-dump'. -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Update of debian packages from 18.2.7
by Bandelow, Gunnar 27 Mar '26

27 Mar '26

Hello, i try to update the ceph packages on debian 12 (bookworm). apt list --upgradable Listing... Done ceph-common/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] cephadm/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] docker-ce-cli/bookworm 5:29.3.1-1~debian.12~bookworm amd64 [upgradable from: 5:29.3.0-1~debian.12~bookworm] docker-ce/bookworm 5:29.3.1-1~debian.12~bookworm amd64 [upgradable from: 5:29.3.0-1~debian.12~bookworm] docker-compose-plugin/bookworm 5.1.1-1~debian.12~bookworm amd64 [upgradable from: 5.1.0-1~debian.12~bookworm] libcephfs2/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] librados2/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] libradosstriper1/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] librbd1/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] librgw2/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-ceph-argparse/stable 18.2.8-1~bpo12+1 all [upgradable from: 18.2.7-1~bpo12+1] python3-ceph-common/stable 18.2.8-1~bpo12+1 all [upgradable from: 18.2.7-1~bpo12+1] python3-cephfs/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-rados/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-rbd/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] python3-rgw/stable 18.2.8-1~bpo12+1 amd64 [upgradable from: 18.2.7-1~bpo12+1] These packages are available for an update. Unfortunately, the ceph-common package seems to depend on the wrong packages version: apt full-upgrade Reading package lists... Done Building dependency tree... Done Reading state information... Done Calculating upgrade... Error! Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: ceph-common : Depends: librbd1 (= 18.2.7-1~bpo12+1) but it is not going to be installed Depends: python3-rbd (= 18.2.7-1~bpo12+1) but it is not going to be installed E: Unable to correct problems, you have held broken packages. Any suggestions? Best regards Gunnar ======================================================= Universitätsrechenzentrum (URZ) Universität Greifswald Felix-Hausdorff-Straße 18 17489 GreifswaldGermany 📞: +49 3834 420 1450 🗨: @mxid-69420:matrix.uni-greifswald.de

1 0

Re: Is it possible to speedup cephadm OSD upgrades?
by Redouane Kachach 26 Mar '26

26 Mar '26

Hi Adam, Short answer is no at this moment. But there's already some good WIP related to that: https://github.com/ceph/ceph/pull/66108 https://github.com/ceph/ceph/pull/67977 Once this new feature is implemented (likely as part of the next Umbrella release) the upgrade process should go faster. Best, Redouane. On Thu, Mar 26, 2026 at 2:26 PM Adam Prycki via ceph-users < ceph-users(a)ceph.io> wrote: > Hello, > > is there a way to speedup OSD redeploy speed during cephadm cluster > upgrade? > > We are currently upgrading our large cluster and going really slowly. > "osd": { > "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) > squid (stable)": 2228, > "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > squid (stable)": 119 > }, > > At current pace upgrade may take a week because cephadm redeploys OSDs > one at a time. > > Is there some way to speed it up? Maybe redeploy OSDs one server at a time? > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Is it possible to speedup cephadm OSD upgrades?
by Adam Prycki 26 Mar '26

26 Mar '26

Hello, is there a way to speedup OSD redeploy speed during cephadm cluster upgrade? We are currently upgrading our large cluster and going really slowly. "osd": { "ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)": 2228, "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 119 }, At current pace upgrade may take a week because cephadm redeploys OSDs one at a time. Is there some way to speed it up? Maybe redeploy OSDs one server at a time? Best regards Adam Prycki

1 0

Agents constantly contacting wrong mgr
by Francesco Di Nucci 26 Mar '26

26 Mar '26

Hi, I'm using Ceph Squid installed with cephadm and I'm having an issue with agents. Cluster health constantly goes in warn as agent do not report to the active mgr - on the other side agents report "Failed to send metadata to mgr: non-200 response <-1> from agent endpoint: [Errno 111] Connection refused". Inspecting agent activity with strace, they're all trying to contact monmgr01.example.com/X.Y.Z.1 , while actually the active manager is monmgr02.example.com/X.Y.Z.1. The issue remains if I redeploy the agents with ceph orch. Any suggestion on how to find out more on the issue? Has it happened to someone else? Thanks -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

1 0

Re: Issues upgrading cephadm mixed intel/ARM64 cluster
by Adam Prycki 25 Mar '26

25 Mar '26

Thank you, I didn't know about this config option. With it disabled now ceph orch upgrade status shows: "target_image": "quay.io/ceph/ceph:v19.2.3", MGR and MON have updated successfully. Cephadm uses the right string for container version. Hopefully we won't have any more issues with this upgrade. Best regards Adam Prycki On 25/03/2026 15:09, Eugen Block via ceph-users wrote: > Hi, > > you could try to disable use_repo_digest: > > ceph config help mgr/cephadm/use_repo_digest > mgr/cephadm/use_repo_digest - Automatically convert image tags to image > digest. Make sure all daemons use the same image > (bool, advanced) > Default: true > Can update at runtime: false > > You would need to restart the MGRs to apply that config update. > > Regards, > Eugen > > Zitat von Adam Prycki via ceph-users <ceph-users(a)ceph.io>: > >> Hello, >> >> we are having issues upgrading from 19.2.2 to 19.2.3 >> >> We have mixed ARM64 AMD64 ceph deployment. >> For performance reason we had to host our MON, MGR and MDS daemons on >> intel server. >> >> It didn't cause any issues for the past 3 years but today we >> discovered that we are not able to perform an upgrade. >> >> Before starting the upgrade, I manually pulled ceph image on all of >> our servers with podman pull. (podman pull quay.io/ceph/ceph:v19.2.3) >> Every server has the new image. >> >> We start upgrade with normal cephadm command: >> ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 >> >> Cephadm always fails on the first host with "UPGRADE_FAILED_PULL: >> Upgrade: failed to pull target image" >> >> ceph orch upgrade status says that ceph performs upgrade to >> "target_image": "quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717", >> not to specified quay.io/ceph/ceph:v19.2.3 >> >> 865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 >> appears to be ARM64 version of the container. https://quay.io/ >> repository/ceph/ceph/manifest/ >> sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee >> >> ARM64 container fails to start on AMD64 machines and upgrade can never >> start. >> >> 2026-03-25 12:27:15,337 7f98685251c0 DEBUG >> -------------------------------------------------------------------------------- >> cephadm ['--image', 'quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495', 'pull'] >> 2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image >> quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717... >> 2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout >> 23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75, >> [quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717] >> 2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from / >> usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host -- >> entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ >> ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 --version >> 2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR >> (catatonit:2): failed to exec pid1: Exec format error >> 2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG >> -------------------------------------------------------------------------------- >> >> Is this a known issue? Has there been a change to how cephadm handles >> image version? >> >> We have made multiple upgrade of this cluster in the past and this is >> the first time we are seeing this issue. >> ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) >> >> Is there a way to force cephadm to use image tags instead if digests. >> >> Best regards >> Adam Prycki >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Issues upgrading cephadm mixed intel/ARM64 cluster
by John Mulligan 25 Mar '26

25 Mar '26

On Wednesday, March 25, 2026 10:01:55 AM Eastern Daylight Time Adam Prycki via ceph-users wrote: > ---- > > Is this a known issue? Has there been a change to how cephadm handles > image version? This is a known issue. I can't find a tracker right now but this has been discussed multiple times in the past and an approach was designed to resolve the issue but AFAIK no one has implemented it. This is not new. I think you are just lucky not to have seen this before. > > We have made multiple upgrade of this cluster in the past and this is > the first time we are seeing this issue. > ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) > > Is there a way to force cephadm to use image tags instead if digests. > Eugen Block has shared the right answer in another reply to your message. Good luck.

1 0

Re: Issues upgrading cephadm mixed intel/ARM64 cluster
by Eugen Block 25 Mar '26

25 Mar '26

Hi, you could try to disable use_repo_digest: ceph config help mgr/cephadm/use_repo_digest mgr/cephadm/use_repo_digest - Automatically convert image tags to image digest. Make sure all daemons use the same image (bool, advanced) Default: true Can update at runtime: false You would need to restart the MGRs to apply that config update. Regards, Eugen Zitat von Adam Prycki via ceph-users <ceph-users(a)ceph.io>: > Hello, > > we are having issues upgrading from 19.2.2 to 19.2.3 > > We have mixed ARM64 AMD64 ceph deployment. > For performance reason we had to host our MON, MGR and MDS daemons > on intel server. > > It didn't cause any issues for the past 3 years but today we > discovered that we are not able to perform an upgrade. > > Before starting the upgrade, I manually pulled ceph image on all of > our servers with podman pull. (podman pull > quay.io/ceph/ceph:v19.2.3) Every server has the new image. > > We start upgrade with normal cephadm command: > ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 > > Cephadm always fails on the first host with "UPGRADE_FAILED_PULL: > Upgrade: failed to pull target image" > > ceph orch upgrade status says that ceph performs upgrade to > "target_image": > "quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717", > not to specified quay.io/ceph/ceph:v19.2.3 > > 865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 > appears to be ARM64 version of the container. > https://quay.io/repository/ceph/ceph/manifest/sha256:af0c5903e901e329adabe2… > > ARM64 container fails to start on AMD64 machines and upgrade can never start. > > 2026-03-25 12:27:15,337 7f98685251c0 DEBUG > -------------------------------------------------------------------------------- > cephadm ['--image', > 'quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495', > 'pull'] > 2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image > quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717... > 2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout > 23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75,[quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717] > 2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from > /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host > --entrypoint ceph --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 > --version > 2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR > (catatonit:2): failed to exec pid1: Exec format error > 2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG > -------------------------------------------------------------------------------- > > Is this a known issue? Has there been a change to how cephadm > handles image version? > > We have made multiple upgrade of this cluster in the past and this > is the first time we are seeing this issue. > ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) > > Is there a way to force cephadm to use image tags instead if digests. > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Issues upgrading cephadm mixed intel/ARM64 cluster
by Adam Prycki 25 Mar '26

25 Mar '26

Hello, we are having issues upgrading from 19.2.2 to 19.2.3 We have mixed ARM64 AMD64 ceph deployment. For performance reason we had to host our MON, MGR and MDS daemons on intel server. It didn't cause any issues for the past 3 years but today we discovered that we are not able to perform an upgrade. Before starting the upgrade, I manually pulled ceph image on all of our servers with podman pull. (podman pull quay.io/ceph/ceph:v19.2.3) Every server has the new image. We start upgrade with normal cephadm command: ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 Cephadm always fails on the first host with "UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" ceph orch upgrade status says that ceph performs upgrade to "target_image": "quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717", not to specified quay.io/ceph/ceph:v19.2.3 865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 appears to be ARM64 version of the container. https://quay.io/repository/ceph/ceph/manifest/sha256:af0c5903e901e329adabe2… ARM64 container fails to start on AMD64 machines and upgrade can never start. 2026-03-25 12:27:15,337 7f98685251c0 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495', 'pull'] 2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717... 2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout 23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75,[quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717] 2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 --version 2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR (catatonit:2): failed to exec pid1: Exec format error 2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG -------------------------------------------------------------------------------- Is this a known issue? Has there been a change to how cephadm handles image version? We have made multiple upgrade of this cluster in the past and this is the first time we are seeing this issue. ( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 ) Is there a way to force cephadm to use image tags instead if digests. Best regards Adam Prycki

1 0