- ceph-users - lists.ceph.io

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 05 Jun '26

05 Jun '26

On Wed, Jun 3, 2026 at 12:42 PM Eugen Block via dev <dev(a)ceph.io> wrote: > > Now that's unexpected, I thought I had misremembered my previous > actions and didn't mention it yet, now I created 3 new VMs, created a > new group, took a group snapshot, all good. But a group snap rollback > is executed although the rbd images have watchers: > > rbd --id openstack group snap create images/test-servers@snap1 > > rbd --id openstack group image ls images/test-servers > images/3df08789-2be9-4e99-9746-9d2edc8c612a_disk > images/7a5d19eb-1034-489f-885a-0074fef59e89_disk > images/f79e323f-87a1-4cb7-ad9c-1108ce73efe3_disk > > > rbd status images/3df08789-2be9-4e99-9746-9d2edc8c612a_disk > Watchers: > watcher=X.X.X.18:0/4236769191 client.<client> cookie=<cookie> > > rbd --id openstack group snap rollback images/test-servers@snap1 > Rolling back to group snapshot: 100% complete...done. > > This shouldn't be possible, I would expect a message like this: > > Rolling back to snapshot: 0% complete...failed. > rbd: rollback failed: (30) Read-only file system > > Is this a known bug? No, this behavior is expected. The reason for why "rbd snap rollback" command can deny the operation with EROFS when the image is mapped in some cases but "rbd group snap rollback" command doesn't do that lies in implementation details (specifically the interaction with exclusive locks on member images). When it comes to rollback operation, the mere presence of a watch isn't really an indicator of anything. That said, I'd recommend shutting down clients before issuing any rollbacks in both standalone image and group scenarios. That way all caches would get invalidated and there is no chance of confusing someone or something with now-stale data. Thanks, Ilya > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > But one more question on this: why is it allowed to remove an image > > from the group if there are existing snapshots? Shouldn't this be > > prevented to keep the group consistency? > > > > And just for my understanding: how are the group snapshots > > technically created? Is that one snapshot for all images or is it an > > individual snapshot per image? > > > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > >> I understand, you're right about the group consistency of course. I > >> just thought if you can remove an image from the group, it would > >> also remove the image's snapshot(s) from the list of snapshots as > >> well. My scenario is: initially I thought it would make sense to > >> have those servers in a group because if I wanted to rollback, it > >> would make sense to do it for all. But then I thought a bit more > >> about it and decided that one of the images actually doesn't make > >> sense to be in that group. Re-adding it will cause more problems in > >> case of rollback... I need to think about this... > >> > >> Thanks a lot for taking the time, I really appreciate it! > >> > >> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >> > >>> On Wed, Jun 3, 2026 at 10:27 AM Eugen Block <eblock(a)nde.ag> wrote: > >>>> > >>>> Hi, > >>>> > >>>> that is correct, log_to_stderr is false in our cluster. And with > >>>> --log-to-stderr true the result is a you expected: > >>>> > >>>> rbd: rollback group to snapshot failed: 2026-06-03T08:06:52.930+0000 > >>>> 7f53896ae0c0 -1 librbd::api::Group: snap_rollback: group snapshot > >>>> membership does not match group membership > >>>> > >>>> But what's the conclusion here? So it's not allowed to rollback if > >>>> memberships don't match. How would I correct the membership? > >>> > >>> Re-add the image back to the group if the image is still around. > >>> > >>>> Because I > >>>> wouldn't want to delete all snapshots from before I removed one image > >>>> from the group. Is there any workaround? > >>> > >>> I'm not sure I see what needs to be worked around here. The group is > >>> supposed to be a logical collection of images where some level of > >>> consistency between images is required, not a random "bag". This > >>> suggests that while images can come and go (i.e. be added and removed > >>> from the group), the group can't always be meaningfully rolled back. > >>> For example, if a group snapshot captured images A, B and C but image > >>> C had since been removed from the group and potentially reformatted, > >>> repurposed for something else or removed altogether, the group's state > >>> exactly as of that snapshot just can't be restored. > >>> > >>> Thanks, > >>> > >>> Ilya > >>> > >>>> > >>>> Thanks, > >>>> Eugen > >>>> > >>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>> > >>>>> On Fri, May 29, 2026 at 6:41 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>>> > >>>>>> The commands were: > >>>>>> > >>>>>> controller02:~# rbd --id user group create images/test-servers > >>>>>> > >>>>>> controller02:~# for i in 0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd > >>>>>> 72f5816c-c1db-44de-b0a2-19d661faa963 > >>>>>> 47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc; do rbd --id user group image add > >>>>>> images/test-servers images/${i}_disk; done > >>>>>> > >>>>>> controller02:~# rbd --id user group image ls images/test-servers > >>>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>>> images/47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc_disk > >>>>>> images/72f5816c-c1db-44de-b0a2-19d661faa963_disk > >>>>>> > >>>>>> controller02:~# rbd --id user group snap create > >>>>>> images/test-servers@snap1 > >>>>>> > >>>>>> controller02:~# rbd --id user group snap ls images/test-servers > >>>>>> NAME STATUS > >>>>>> snap1 ok > >>>>>> > >>>>>> > >>>>>> # rollback works for all images > >>>>>> controller02:~# rbd --id user group snap rollback > >>>>>> images/test-servers@snap1 > >>>>>> Rolling back to group snapshot: 100% complete...done. > >>>>>> > >>>>>> # removing one image from the group > >>>>>> controller02:~# rbd --id user group image rm images/test-servers > >>>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>>> > >>>>>> # rollback fails > >>>>>> controller02:~# rbd --id user group snap rollback > >>>>>> images/test-servers@snap1 > >>>>>> Rolling back to group snapshot: 0% complete...failed. > >>>>>> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>>> > >>>>>> I'll add the debug output later, will need to sanitze it first. But I > >>>>>> don't see anything obvious in there. > >>>>> > >>>>> Hi Eugen, > >>>>> > >>>>> Based on the above, it's https://tracker.ceph.com/issues/66300 and is > >>>>> therefore the intended behavior. The only fly in the ointment is that > >>>>> you aren't seeing the associated "group snapshot membership does not > >>>>> match group membership" error message. > >>>>> > >>>>> You not seeing it is consistent with the attached debug output where > >>>>> only very early messenger traffic is present and nothing beyond that. > >>>>> It suggests some non-conventional settings in the cluster-wide config > >>>>> such as log_to_stderr being set to false or similar. > >>>>> > >>>>> Can you try appending --log-to-stderr true to "rbd group snap rollback" > >>>>> command? > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Ilya > >>>>> > >>>>>> > >>>>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>>> > >>>>>>> On Fri, May 29, 2026 at 4:05 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> thanks for your quick reply. No I didn't see any additional output > >>>>>>>> than the one I shared (invalid argument). I could add debug log level > >>>>>>>> if necessary. > >>>>>>> > >>>>>>> That error message should have been displayed no matter the log level, > >>>>>>> so something other than https://tracker.ceph.com/issues/66300 might be > >>>>>>> involved. > >>>>>>> > >>>>>>> What exactly do you mean by "I removed an image from the group > >>>>>>> snapshot"? Which commands were run there and in what order? > >>>>>>> > >>>>>>>> But one more detail, I also tried the rollback directly within the > >>>>>>>> cephadm shell (so version 19.2.3) with the same result: > >>>>>>>> > >>>>>>>> ceph03:~ # cephadm shell > >>>>>>>> ... > >>>>>>>> [ceph: root@ceph03 /]# rbd group snap rollback > >>>>>>>> images/test-servers@20260430_start > >>>>>>>> Rolling back to group snapshot: 0% complete...failed. > >>>>>>>> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>>>>> > >>>>>>>> [ceph: root@ceph03 /]# ceph -v > >>>>>>>> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > >>>>>>>> squid (stable) > >>>>>>> > >>>>>>> Can you try appending --debug-ms 1 --debug-rbd 20 to the command > >>>>>>> (let's stick to this cephadm shell) and attach the output? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> > >>>>>>> Ilya > >>>>>>> > >>>>>>>> > >>>>>>>> Thanks! > >>>>>>>> Eugen > >>>>>>>> > >>>>>>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>>>>> > >>>>>>>> > On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users > >>>>>>>> > <ceph-users(a)ceph.io> wrote: > >>>>>>>> >> > >>>>>>>> >> Hi, > >>>>>>>> >> > >>>>>>>> >> I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > >>>>>>>> >> version 19.2.1), the Ceph cluster version is 19.2.3. The > >>>>>>>> client fails > >>>>>>>> >> with "invalid argument": > >>>>>>>> >> > >>>>>>>> >> controller02:~# rbd --id <user> group snap rollback --pool images > >>>>>>>> >> --group test-servers --snap 20260430_start > >>>>>>>> >> Rolling back to group snapshot: 0% complete...failed. > >>>>>>>> >> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>>>>> >> > >>>>>>>> >> controller02:~# ceph -v > >>>>>>>> >> ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) > >>>>>>>> >> squid (stable) > >>>>>>>> >> > >>>>>>>> >> But running the same command (just as admin not as <user>) > >>>>>>>> on a Ceph > >>>>>>>> >> node works: > >>>>>>>> >> > >>>>>>>> >> ceph03:~ # rbd group snap rollback --pool images --group > >>>>>>>> test-servers > >>>>>>>> >> --snap 20260430_start > >>>>>>>> >> Rolling back to group snapshot: 100% complete...done. > >>>>>>>> >> > >>>>>>>> >> ceph03:~ # ceph -v > >>>>>>>> >> ceph version 16.2.13-66-g54799ee0666 > >>>>>>>> >> (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> What seems to be the issue here is that I removed an image from the > >>>>>>>> >> group snapshot. I wonder if it could be this bug [0] which > >>>>>>>> is supposed > >>>>>>>> >> to be fixed in 19.2.0 according to the "Released In" field of the > >>>>>>>> >> Squid backport tracker [1]. > >>>>>>>> >> > >>>>>>>> >> This seems a little inconsistent to me, could someone > >>>>>>>> please clarify? > >>>>>>>> > > >>>>>>>> > Hi Eugen, > >>>>>>>> > > >>>>>>>> > Did you see "group snapshot membership does not match group > >>>>>>>> membership" > >>>>>>>> > error message when the rollback command failed with 19.2.1 client? > >>>>>>>> > > >>>>>>>> > Thanks, > >>>>>>>> > > >>>>>>>> > Ilya > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >>>> > >

2 3

Re: Best "modern" approach to drain/fill a OSD server
by Janne Johansson 05 Jun '26

05 Jun '26

> If it can be useful, I'm thinking of organizing an OSD balancing happy > hour call in the next month or so. > Maybe mid-May or so on a Friday afternoon (EU time) / morning (US time). Also interested. -- May the most significant bit of your life be positive.

3 2

Re: [EXT] Crash on newly created NVMe+HDD OSD
by Justin Mammarella 05 Jun '26

05 Jun '26

Hi Jan, We encountered this issue after upgrading to Tentacle 20.2.1 and attempting to rebuild some of our OSDs that use an NVMe device for the WAL/DB. I was unable to find any instance of this issue being reported upstream, either via a GitHub pull request or a bug report. My colleague has submitted a bug report here with the code regression, including a workaround: https://tracker.ceph.com/issues/77127 Justin Mammarella Storage Infrastructure Lead Research Computing Services The University of Melbourne Victoria 3010, Australia From: Jan Kasprzak via ceph-users <ceph-users(a)ceph.io> Date: Monday, 30 March 2026 at 6:17 pm To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [EXT] [ceph-users] Crash on newly created NVMe+HDD OSD External email: Please exercise caution Hello, Ceph users, I wanted to replace a HDD with bad blocks in an OSD which has block.db on NVMe. Removing the OSD (ceph-volume lvm zap, etc.) went well, no VG/LV remained for data or metadata. But after replacing the HDD, preparing a new HDD+NVMe OSD, and starting it with "ceph-volume lvm activate --all", the OSD daemon crashed with terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input Retrying the proces once more I noticed that ceph-volume created a VG/PV/LV on the HDD (/dev/sda), but did not create a VG/PV/LV on /dev/nvme1n1p4. Zapping the result once more and creating a HDD-only OSD helped, the OSD daemon started and joined the cluster correctly. I then stopped it once more, and tried to create a db volume myself, and migrate db to it: systemctl stop ceph-osd@$OSD_ID NEW_DB_VG=ceph-`uuidgen` NEW_DB_LV=osd-db-`uuidgen` vgcreate $NEW_DB_VG /dev/nvme1n1p4 lvcreate -a y -l 100%FREE -n $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm new-db --osd-id $OSD_ID --osd-fsid $OSD_FSID --target $NEW_DB_VG/$NEW_DB_LV ceph-volume lvm migrate --osd-id $OSD_ID --osd-fsid $OSD_FSID --from data --target $NEW_DB_VG/$NEW_DB_LV systemctl start ceph-osd@$OSD_ID ... and it started correctly and did not crash. I used the same commands previously on Ceph 19 when replacing HDDs, and it worked without problem. This is the first time I replaced a HDD after upgrading to Ceph 20.2.0, so it might be a regression in ceph-volume of 20.2.0. Below is the output of "ceph-volume lvm prepare", and the crash report from journal. Did I do something incorrect here? Thanks! -Yenya # ceph-volume lvm prepare --bluestore --block.db /dev/nvme1n1p4 --data /dev/sda Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: vgcreate --force --yes ceph-8805a195-57fd-4054-8282-a307d15955a5 /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-8805a195-57fd-4054-8282-a307d15955a5" successfully created Running command: lvcreate --yes -l 2861055 -n osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 ceph-8805a195-57fd-4054-8282-a307d15955a5 stdout: Logical volume "osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963" created. Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-25 Running command: /usr/sbin/restorecon /var/lib/ceph/osd/ceph-25 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 Running command: /usr/bin/ln -s /dev/ceph-8805a195-57fd-4054-8282-a307d15955a5/osd-block-db38dbfb-e161-4680-93e3-ba4abaa36963 /var/lib/ceph/osd/ceph-25/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-25/activate.monmap stderr: got monmap epoch 39 --> Creating keyring file for osd.25 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/keyring Running command: /usr/bin/chown -R ceph:ceph /dev/nvme1n1p4 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-25/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 25 --monmap /var/lib/ceph/osd/ceph-25/activate.monmap --keyfile - --bluestore-block-db-path /dev/nvme1n1p4 --osd-data /var/lib/ceph/osd/ceph-25/ --osd-uuid db38dbfb-e161-4680-93e3-ba4abaa36963 --setuser ceph --setgroup ceph stderr: 2026-03-30T07:59:06.150+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25//block) No valid bdev label found stderr: 2026-03-30T07:59:06.175+0200 7f2ae8102900 -1 bluestore(/var/lib/ceph/osd/ceph-25/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/sda Mar 30 07:52:31 osdhost systemd[1]: Starting Ceph object storage daemon osd.25... Mar 30 07:52:31 osdhost systemd[1]: Started Ceph object storage daemon osd.25. Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: terminate called after throwing an instance of 'ceph::buffer::v15_2_0::malformed_input' Mar 30 07:52:31 osdhost ceph-osd[1140285]: what(): static void bluefs_fnode_t::_denc_finish(ceph::buffer::v15_2_0::ptr::const_iterator&, char**, uint32_t*): Malformed input [buffer:3] Mar 30 07:52:31 osdhost ceph-osd[1140285]: *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:31 osdhost ceph-osd[1140285]: -51> 2026-03-30T07:52:31.738+0200 7f2dc400c900 -1 Falling back to public interface Mar 30 07:52:31 osdhost ceph-osd[1140285]: 0> 2026-03-30T07:52:31.931+0200 7f2dc400c900 -1 *** Caught signal (Aborted) ** Mar 30 07:52:31 osdhost ceph-osd[1140285]: in thread 7f2dc400c900 thread_name:ceph-osd Mar 30 07:52:31 osdhost ceph-osd[1140285]: ceph version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable - RelWithDebInfo) Mar 30 07:52:31 osdhost ceph-osd[1140285]: 1: /lib64/libc.so.6(+0x3fc30) [0x7f2dc3e3fc30] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 2: /lib64/libc.so.6(+0x8d02c) [0x7f2dc3e8d02c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 3: raise() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 4: abort() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f2dc4ca1b21] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 6: /lib64/libstdc++.so.6(+0xad53c) [0x7f2dc4cad53c] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 7: /lib64/libstdc++.so.6(+0xad5a7) [0x7f2dc4cad5a7] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 8: /lib64/libstdc++.so.6(+0xad809) [0x7f2dc4cad809] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 9: /usr/bin/ceph-osd(+0x45ceef) [0x558458ee6eef] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 10: (bluefs_super_t::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)+0x1a0) [0x55845962f140] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 11: (BlueFS::_open_super()+0xe6) [0x5584596123a6] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 12: (BlueFS::mount()+0xc2) [0x5584595fbf42] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 13: (BlueStore::_open_bluefs(bool, bool)+0x294) [0x55845953ed34] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 14: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x5d0) [0x55845953f8b0] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 15: (BlueStore::_open_db(bool, bool, bool)+0x135) [0x5584595407b5] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 16: (BlueStore::_open_db_and_around(bool, bool)+0x31a) [0x55845954137a] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 17: (BlueStore::_mount()+0x281) [0x558459546a31] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 18: (OSD::init()+0x509) [0x5584590bea09] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 19: main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 20: /lib64/libc.so.6(+0x2a610) [0x7f2dc3e2a610] Mar 30 07:52:31 osdhost ceph-osd[1140285]: 21: __libc_start_main() Mar 30 07:52:31 osdhost ceph-osd[1140285]: 22: _start() Mar 30 07:52:31 osdhost ceph-osd[1140285]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Main process exited, code=dumped, status=6/ABRT Mar 30 07:52:32 osdhost systemd[1]: ceph-osd(a)25.service: Failed with result 'core-dump'. -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: coe, an adventure story
by George Shuklin 05 Jun '26

05 Jun '26

On 6/4/26 1:33 AM, C U via ceph-users wrote: > Enjoy! Lessons learned and demonstration of "AI enhanced" infrastructure. > > TL:DR used ceph-kvstore-tool bluestore-k work around a data corruption > issue cause by machine operator. > > https://github.com/tu503/coe/blob/main/2026-06-03-ceph-bulk-rbd-rm/COE.md This is the fine piece of reading, thank you. There are few things which are not clear to me: 1. If osd fails to start after abort, this is a bug. There is no link to issue anywhere in the report. Was bug identified, or at least reported? 2. It says that controller was overloaded. At the same time, there was a notice on hard drives behind controller. Are you sure it was controller overloaded, and not a normal drive thrashing? Was you able to reproduce this with synthetic load? Was was ratio between drives and ports (and of what kind?) in your system?

1 0

Re: tentacle 20.2.2 QE validation status
by Yuri Weinstein 04 Jun '26

04 Jun '26

My apologies typo for orch dev lead name: I am still seeking approvals for: fs: Venky orch/ceph-volume: Redo & Guillaume On Thu, Jun 4, 2026 at 1:37 PM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > I am still seeking approvals for: > > fs: Venky > orch/ceph-volume: Adam King & Guillaume > > We have an outstanding PRs tracker https://tracker.ceph.com/issues/77086, > that need to be approved and PRs merged before we can finish testing > this release. > > We also plan for the old LRC to be upgraded to use this release. > > TIA > > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75878#note-6 >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya >> nvmeof: Aviv, Vallari >> >> upgrade: Patrick >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA >> >

1 0

coe, an adventure story
by C U 03 Jun '26

03 Jun '26

Enjoy! Lessons learned and demonstration of "AI enhanced" infrastructure. TL:DR used ceph-kvstore-tool bluestore-k work around a data corruption issue cause by machine operator. https://github.com/tu503/coe/blob/main/2026-06-03-ceph-bulk-rbd-rm/COE.md regards,

1 0

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 03 Jun '26

03 Jun '26

On Wed, Jun 3, 2026 at 12:01 PM Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > But one more question on this: why is it allowed to remove an image > from the group if there are existing snapshots? Shouldn't this be > prevented to keep the group consistency? It wasn't prevented when groups were originally introduced and was considered to be too restrictive for a retroactive fix. Removing an image from the group doesn't invalidate any group snapshots -- the corresponding image snapshots remain there. An image can always be re-added to the group and as long as the operator is OK with the HEAD of the image getting rolled back after the image was potentially used for something else while outside the group, "rbd group snap rollback" command can be run and would work as expected. > > And just for my understanding: how are the group snapshots technically > created? Is that one snapshot for all images or is it an individual > snapshot per image? The latter -- an individual snapshot per member image. You can see these snapshots if you pass --all to "rbd snap ls" command. Thanks, Ilya > > Zitat von Eugen Block <eblock(a)nde.ag>: > > > I understand, you're right about the group consistency of course. I > > just thought if you can remove an image from the group, it would > > also remove the image's snapshot(s) from the list of snapshots as > > well. My scenario is: initially I thought it would make sense to > > have those servers in a group because if I wanted to rollback, it > > would make sense to do it for all. But then I thought a bit more > > about it and decided that one of the images actually doesn't make > > sense to be in that group. Re-adding it will cause more problems in > > case of rollback... I need to think about this... > > > > Thanks a lot for taking the time, I really appreciate it! > > > > Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > > > >> On Wed, Jun 3, 2026 at 10:27 AM Eugen Block <eblock(a)nde.ag> wrote: > >>> > >>> Hi, > >>> > >>> that is correct, log_to_stderr is false in our cluster. And with > >>> --log-to-stderr true the result is a you expected: > >>> > >>> rbd: rollback group to snapshot failed: 2026-06-03T08:06:52.930+0000 > >>> 7f53896ae0c0 -1 librbd::api::Group: snap_rollback: group snapshot > >>> membership does not match group membership > >>> > >>> But what's the conclusion here? So it's not allowed to rollback if > >>> memberships don't match. How would I correct the membership? > >> > >> Re-add the image back to the group if the image is still around. > >> > >>> Because I > >>> wouldn't want to delete all snapshots from before I removed one image > >>> from the group. Is there any workaround? > >> > >> I'm not sure I see what needs to be worked around here. The group is > >> supposed to be a logical collection of images where some level of > >> consistency between images is required, not a random "bag". This > >> suggests that while images can come and go (i.e. be added and removed > >> from the group), the group can't always be meaningfully rolled back. > >> For example, if a group snapshot captured images A, B and C but image > >> C had since been removed from the group and potentially reformatted, > >> repurposed for something else or removed altogether, the group's state > >> exactly as of that snapshot just can't be restored. > >> > >> Thanks, > >> > >> Ilya > >> > >>> > >>> Thanks, > >>> Eugen > >>> > >>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>> > >>>> On Fri, May 29, 2026 at 6:41 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>> > >>>>> The commands were: > >>>>> > >>>>> controller02:~# rbd --id user group create images/test-servers > >>>>> > >>>>> controller02:~# for i in 0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd > >>>>> 72f5816c-c1db-44de-b0a2-19d661faa963 > >>>>> 47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc; do rbd --id user group image add > >>>>> images/test-servers images/${i}_disk; done > >>>>> > >>>>> controller02:~# rbd --id user group image ls images/test-servers > >>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>> images/47d6144e-0d5a-4dc7-82dd-5be3edf9f6cc_disk > >>>>> images/72f5816c-c1db-44de-b0a2-19d661faa963_disk > >>>>> > >>>>> controller02:~# rbd --id user group snap create images/test-servers@snap1 > >>>>> > >>>>> controller02:~# rbd --id user group snap ls images/test-servers > >>>>> NAME STATUS > >>>>> snap1 ok > >>>>> > >>>>> > >>>>> # rollback works for all images > >>>>> controller02:~# rbd --id user group snap rollback > >>>>> images/test-servers@snap1 > >>>>> Rolling back to group snapshot: 100% complete...done. > >>>>> > >>>>> # removing one image from the group > >>>>> controller02:~# rbd --id user group image rm images/test-servers > >>>>> images/0f69278e-00c2-46b0-b6e7-0b06e9c8b6fd_disk > >>>>> > >>>>> # rollback fails > >>>>> controller02:~# rbd --id user group snap rollback > >>>>> images/test-servers@snap1 > >>>>> Rolling back to group snapshot: 0% complete...failed. > >>>>> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>> > >>>>> I'll add the debug output later, will need to sanitze it first. But I > >>>>> don't see anything obvious in there. > >>>> > >>>> Hi Eugen, > >>>> > >>>> Based on the above, it's https://tracker.ceph.com/issues/66300 and is > >>>> therefore the intended behavior. The only fly in the ointment is that > >>>> you aren't seeing the associated "group snapshot membership does not > >>>> match group membership" error message. > >>>> > >>>> You not seeing it is consistent with the attached debug output where > >>>> only very early messenger traffic is present and nothing beyond that. > >>>> It suggests some non-conventional settings in the cluster-wide config > >>>> such as log_to_stderr being set to false or similar. > >>>> > >>>> Can you try appending --log-to-stderr true to "rbd group snap rollback" > >>>> command? > >>>> > >>>> Thanks, > >>>> > >>>> Ilya > >>>> > >>>>> > >>>>> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>> > >>>>> > On Fri, May 29, 2026 at 4:05 PM Eugen Block <eblock(a)nde.ag> wrote: > >>>>> >> > >>>>> >> Hi, > >>>>> >> > >>>>> >> thanks for your quick reply. No I didn't see any additional output > >>>>> >> than the one I shared (invalid argument). I could add debug log level > >>>>> >> if necessary. > >>>>> > > >>>>> > That error message should have been displayed no matter the log level, > >>>>> > so something other than https://tracker.ceph.com/issues/66300 might be > >>>>> > involved. > >>>>> > > >>>>> > What exactly do you mean by "I removed an image from the group > >>>>> > snapshot"? Which commands were run there and in what order? > >>>>> > > >>>>> >> But one more detail, I also tried the rollback directly within the > >>>>> >> cephadm shell (so version 19.2.3) with the same result: > >>>>> >> > >>>>> >> ceph03:~ # cephadm shell > >>>>> >> ... > >>>>> >> [ceph: root@ceph03 /]# rbd group snap rollback > >>>>> >> images/test-servers@20260430_start > >>>>> >> Rolling back to group snapshot: 0% complete...failed. > >>>>> >> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>> >> > >>>>> >> [ceph: root@ceph03 /]# ceph -v > >>>>> >> ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) > >>>>> >> squid (stable) > >>>>> > > >>>>> > Can you try appending --debug-ms 1 --debug-rbd 20 to the command > >>>>> > (let's stick to this cephadm shell) and attach the output? > >>>>> > > >>>>> > Thanks, > >>>>> > > >>>>> > Ilya > >>>>> > > >>>>> >> > >>>>> >> Thanks! > >>>>> >> Eugen > >>>>> >> > >>>>> >> Zitat von Ilya Dryomov <idryomov(a)gmail.com>: > >>>>> >> > >>>>> >> > On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users > >>>>> >> > <ceph-users(a)ceph.io> wrote: > >>>>> >> >> > >>>>> >> >> Hi, > >>>>> >> >> > >>>>> >> >> I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > >>>>> >> >> version 19.2.1), the Ceph cluster version is 19.2.3. The > >>>>> client fails > >>>>> >> >> with "invalid argument": > >>>>> >> >> > >>>>> >> >> controller02:~# rbd --id <user> group snap rollback --pool images > >>>>> >> >> --group test-servers --snap 20260430_start > >>>>> >> >> Rolling back to group snapshot: 0% complete...failed. > >>>>> >> >> rbd: rollback group to snapshot failed: (22) Invalid argument > >>>>> >> >> > >>>>> >> >> controller02:~# ceph -v > >>>>> >> >> ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) > >>>>> >> >> squid (stable) > >>>>> >> >> > >>>>> >> >> But running the same command (just as admin not as <user>) > >>>>> on a Ceph > >>>>> >> >> node works: > >>>>> >> >> > >>>>> >> >> ceph03:~ # rbd group snap rollback --pool images --group > >>>>> test-servers > >>>>> >> >> --snap 20260430_start > >>>>> >> >> Rolling back to group snapshot: 100% complete...done. > >>>>> >> >> > >>>>> >> >> ceph03:~ # ceph -v > >>>>> >> >> ceph version 16.2.13-66-g54799ee0666 > >>>>> >> >> (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > >>>>> >> >> > >>>>> >> >> > >>>>> >> >> What seems to be the issue here is that I removed an image from the > >>>>> >> >> group snapshot. I wonder if it could be this bug [0] which > >>>>> is supposed > >>>>> >> >> to be fixed in 19.2.0 according to the "Released In" field of the > >>>>> >> >> Squid backport tracker [1]. > >>>>> >> >> > >>>>> >> >> This seems a little inconsistent to me, could someone > >>>>> please clarify? > >>>>> >> > > >>>>> >> > Hi Eugen, > >>>>> >> > > >>>>> >> > Did you see "group snapshot membership does not match group > >>>>> membership" > >>>>> >> > error message when the rollback command failed with 19.2.1 client? > >>>>> >> > > >>>>> >> > Thanks, > >>>>> >> > > >>>>> >> > Ilya > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> > >>>>> > >>>>> > >>> > >>> > >>> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: RBD: group snap rollback invalid argument
by Ilya Dryomov 03 Jun '26

03 Jun '26

On Fri, May 29, 2026 at 2:33 PM Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > I wanted to rollback a group snapshot on Ubuntu 24.04 (rbd client > version 19.2.1), the Ceph cluster version is 19.2.3. The client fails > with "invalid argument": > > controller02:~# rbd --id <user> group snap rollback --pool images > --group test-servers --snap 20260430_start > Rolling back to group snapshot: 0% complete...failed. > rbd: rollback group to snapshot failed: (22) Invalid argument > > controller02:~# ceph -v > ceph version 19.2.1 (9efac4a81335940925dd17dbf407bfd6d3860d28) squid (stable) > > But running the same command (just as admin not as <user>) on a Ceph > node works: > > ceph03:~ # rbd group snap rollback --pool images --group test-servers > --snap 20260430_start > Rolling back to group snapshot: 100% complete...done. > > ceph03:~ # ceph -v > ceph version 16.2.13-66-g54799ee0666 > (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) > > > What seems to be the issue here is that I removed an image from the > group snapshot. I wonder if it could be this bug [0] which is supposed > to be fixed in 19.2.0 according to the "Released In" field of the > Squid backport tracker [1]. > > This seems a little inconsistent to me, could someone please clarify? Hi Eugen, Did you see "group snapshot membership does not match group membership" error message when the rollback command failed with 19.2.1 client? Thanks, Ilya

2 10

Re: The mail threading on this list is broken
by Frédéric Nass 03 Jun '26

03 Jun '26

That's great news! Thank you, Adam, for letting everyone know. We're looking forward to it. Best regards, Frédéric -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass(a)clyso.com Le dim. 31 mai 2026 à 16:22, Adam Kraitman via ceph-users < ceph-users(a)ceph.io> a écrit : > Thank you for the detailed analysis, the threading issue you've described > is a known problem with the current mailing list infrastructure. We're > planning to migrate all mailing lists to Groups.io, and this issue will be > resolved as part of that migration. > > We'll share more details and the migration timeline once everything is > finalized > > On Sun, May 31, 2026 at 3:08 PM Kai Stian Olstad via ceph-users < > ceph-users(a)ceph.io> wrote: > > > Hi > > > > For several month the mail threading on this list broken and making the > > list hard to read. > > It can easily be seen by looking at the archive, in this link for about > > 9 mount(as of writing) > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > > > > Subjects all have nice subject without "Re:" for replay, and on the > > right you see how may people(person icon) in the thread and number of > > post(talking bubble) in the thread. > > > > But if we look at more recent posts like this one > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > > you see a lot of "Re:" in the subject and most of them have one person > > and only one post because the threading is broken. > > > > Take this recent mail "Weird behavior for 2 OSDs in our cluster" > > The fist mail by Wannes Smet > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/62B63BH33F3… > > when i check my mail client this mail has following ID. > > "Message-ID: > > < > > > 0100019e7c77595f-ffb4d04b-1771-46cf-8b60-10795bc6a1c5-000000(a)email.amazonses.com > > >" > > > > The reply from Eugen Block > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/E63Z3JR7DMS… > > the header says this is a reply to different ID > > "In-Reply-To: > > < > > > 0100019e7c7979e5-8d4d96e6-1c93-4a76-a5cd-efdf25ada853-000000(a)email.amazonses.com > > >" > > > > When I search the header of the mail from Wannes Smet I find no such ID > > that is in the In-Reply-To. > > > > It looks to me that the mailing-list server is sending different > > "Messages-ID" to the members on the list. > > This should be the same or the threading breaks as we can se. > > > > Can this be fixed please? > > > > -- > > Kai Stian Olstad > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

test requested y Anthony Middleton
by zouhair nador 02 Jun '26

02 Jun '26

Hello [cid:d40f5272-f460-403e-be1b-ca2561377aed]

1 1