- ceph-users - lists.ceph.io

[RGW] Setup 2 zones within a cluster does not sync data
by Huy Nguyen 30 Jul '24

30 Jul '24

Hi, Currently I'm testing replication ability between 2 zones in the same cluster. But only metadata is synced, not the data. I checked endpoints, system_key,... all good. If anyone have any idea, please guide me to resolve this situation. Thanks The radosgw shows this log on both side, primary and secondary: rgw rados thread: ERROR: failed to run sync ... RGW-SYNC:data:sync:shard[54]: ERROR: failed to read remote data log info: ret=-5 RGW-SYNC:data:sync:shard[90]: ERROR: failed to read remote data log info: ret=-5 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5 Here is my configure (it might be long): zonegroup hn which contains hn-1 and hn-2, each zone have its own endpoint and radosgw instance on two different node: { "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281", "epoch": 4, "predecessor_uuid": "e1981515-7355-4029-b26c-6be3de943be0", "sync_status": [], "period_map": { "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281", "zonegroups": [ { "id": "9ced5b3b-5322-4ec9-862c-521bfff85c91", "name": "hn", "api_name": "hn", "is_master": true, "endpoints": [ "http://10.3.55.66:8080" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "zones": [ { "id": "0c5aac15-c899-46aa-bff9-353bf17f9602", "name": "hn-2", "endpoints": [ "http://10.3.53.107:8080" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] }, { "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "name": "hn-1", "endpoints": [ "http://10.3.55.66:8080" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] } ], "short_zone_ids": [ { "key": "0c5aac15-c899-46aa-bff9-353bf17f9602", "val": 3534260942 }, { "key": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "val": 1656529937 } ] }, "master_zonegroup": "9ced5b3b-5322-4ec9-862c-521bfff85c91", "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, ...ratelimit... }, "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "realm_name": "ss-realm", "realm_epoch": 2 } zone hn-1: { "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "name": "hn-1", "domain_root": "hn-1.rgw.meta:root", "control_pool": "hn-1.rgw.control", "gc_pool": "hn-1.rgw.log:gc", "lc_pool": "hn-1.rgw.log:lc", "log_pool": "hn-1.rgw.log", "intent_log_pool": "hn-1.rgw.log:intent", "usage_log_pool": "hn-1.rgw.log:usage", "roles_pool": "hn-1.rgw.meta:roles", "reshard_pool": "hn-1.rgw.log:reshard", "user_keys_pool": "hn-1.rgw.meta:users.keys", "user_email_pool": "hn-1.rgw.meta:users.email", "user_swift_pool": "hn-1.rgw.meta:users.swift", "user_uid_pool": "hn-1.rgw.meta:users.uid", "otp_pool": "hn-1.rgw.otp", "system_key": { "access_key": "LIW8M2OTOCOHA41OJUGO", "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "hn-1.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "hn-1.rgw.buckets.data" } }, "data_extra_pool": "hn-1.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "notif_pool": "hn-1.rgw.log:notif" } zone hn-2: { "id": "0c5aac15-c899-46aa-bff9-353bf17f9602", "name": "hn-2", "domain_root": "hn-2.rgw.meta:root", "control_pool": "hn-2.rgw.control", "gc_pool": "hn-2.rgw.log:gc", "lc_pool": "hn-2.rgw.log:lc", "log_pool": "hn-2.rgw.log", "intent_log_pool": "hn-2.rgw.log:intent", "usage_log_pool": "hn-2.rgw.log:usage", "roles_pool": "hn-2.rgw.meta:roles", "reshard_pool": "hn-2.rgw.log:reshard", "user_keys_pool": "hn-2.rgw.meta:users.keys", "user_email_pool": "hn-2.rgw.meta:users.email", "user_swift_pool": "hn-2.rgw.meta:users.swift", "user_uid_pool": "hn-2.rgw.meta:users.uid", "otp_pool": "hn-2.rgw.otp", "system_key": { "access_key": "LIW8M2OTOCOHA41OJUGO", "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "hn-2.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "hn-2.rgw.buckets.data" } }, "data_extra_pool": "hn-2.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "notif_pool": "hn-2.rgw.log:notif" }

3 8

Documentation for meaning of "tag cephfs" in OSD caps
by Petr Bena 29 Jul '24

29 Jul '24

Hello In https://docs.ceph.com/en/latest/cephfs/client-auth/ we can find that ceph fs authorize cephfs_a client.foo / r /bar rw Results in client.foo key: *key* caps: [mds] allow r, allow rw path=/bar caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs_a What is this "tag cephfs" thing? It seems like some undocumented black magic to me, since I can't find anything that documents it. Can someone explain how it works under the hood? What does it expand to? What does it limit and how?

4 3

0 slow ops message stuck for down+out OSD
by Frank Schilder 29 Jul '24

29 Jul '24

Hi all, we had a failing disk (with slow ops) and I shut down the OSD. Its status down+out. However, I still see this message stuck in the output of ceph status and ceph health detail: 0 slow ops, oldest one blocked for 70 sec, osd.183 has slow ops I believe there was a case about that some time ago, but I can't find it. How can I get rid of this stuck warning? Our cluster is octopus latest. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

snaptrim not making progress
by Frank Schilder 29 Jul '24

29 Jul '24

Hi all, our cluster is octopus latest. We seem to have a problem with snaptrim. On a pool for HDD RBD images I observed today that all PGs are either in state snaptrim or snaptrim_wait. It looks like the snaptrim process does not actually make any progress. There is no CPU activity by these OSDs indicating they would do snaptrimming (usuallu they would at least use 50% CPU as shown in top). I also don't see anything in the OSD logs. For our VMs we run daily snapshot rotation and snaptrim usually finishes within a few minutes. We had a VM with disks on that pool cause an error due to a hanging virsh domfsfreeze command. This, however, is routine, we see this happening every now and then without any follow-up issues. I'm wondering now if we might have hit a race for the first time. Is there anything on an RBD image or pool that could block snaptrim from starting or progressing? Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 2

v18.2.4 Reef released
by Yuri Weinstein 29 Jul '24

29 Jul '24

We're happy to announce the 4th release in the Reef series. An early build of this release was accidentally exposed and packaged as 18.2.3 by the Debian project in April. That 18.2.3 release should not be used. The official release was re-tagged as v18.2.4 to avoid further confusion. v18.2.4 container images, now based on CentOS 9, may be incompatible on older kernels (e.g., Ubuntu 18.04) due to differences in thread creation methods. Users upgrading to v18.2.4 container images with older OS versions may encounter crashes during `pthread_create`. For workarounds, refer to the related tracker. However, we recommend upgrading your OS to avoid this unsupported combination. Related tracker: https://tracker.ceph.com/issues/66989 We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2024/v18-2-4-reef-released/ Notable Changes --------------- * RADOS: This release fixes a bug (https://tracker.ceph.com/issues/61948) where pre-reef clients were allowed to connect to the `pg-upmap-primary` (https://docs.ceph.com/en/reef/rados/operations/read-balancer/) interface despite users having set `require-min-compat-client=reef`, leading to an assert in the osds and mons. You are susceptible to this bug in reef versions prior to 18.2.3 if 1) you are using an osdmap generated via the offline osdmaptool with the `--read` option or 2) you have explicitly generated pg-upmap-primary mappings with the CLI command. Please note that the fix is minimal and does not address corner cases such as adding a mapping in the middle of an upgrade or in a partially upgraded cluster (related trackers linked in https://tracker.ceph.com/issues/61948) As such, we recommend removing any existing pg-upmap-primary mappings until remaining issues are addressed in future point releases. See https://tracker.ceph.com/issues/61948#note-32 for instructions on how to remove existing pg-upmap-primary mappings. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to being prone to false negative results. It's safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: The option ``--image-id`` has been added to `rbd children` CLI command, so it can be run for images in the trash. Related tracker: https://tracker.ceph.com/issues/65393 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.4.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: e7ad5345525c7aa95470c26863873b581076945d

7 9

How to specify id on newly created OSD with Ceph Orchestrator
by Iztok Gregori 29 Jul '24

29 Jul '24

Hi, Yesterday I had to replace a couple of failed/failing OSDs using Ceph Orchestrator (on Ceph 18.2.1), but I've made a mistake and I removed them without the "--replace" option. I wasn't able to find the correct option (if any) to specify an id for newly created OSD. The closest one was "osd_id_claims", but if I understood correctly is used to claim an OSD id which is still in the crush map (as destroyed). Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation? Cheers Iztok P.S. At the end I (re)added the OSD in the correct order, starting with the lowest id device. -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu

4 6

Stuck in remapped state?
by Tim Holloway 27 Jul '24

27 Jul '24

I was in the middle of tuning my OSDs when lightning blew me off the Internet. Had to wait 5 days for my ISP to send a tech and replace a fried cable. In the mean time, among other things. I had some serious time drift between servers thanks to the OS upgrades replacing NTP with chrony and me not having thought to re-establish a master in-house timeserver. Ceph tried really hard to keep up with all that, but eventually it was just too much. Now I've got an offline filesystem and apparently it's stuck trying to get back online again. The forensics: [ceph: root@www7 /]# ceph -s cluster: id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f health: HEALTH_WARN failed to probe daemons or devices 1 filesystem is degraded 1/3 mons down, quorum www7,ceph03 services: mon: 3 daemons, quorum www7,ceph03 (age 2m), out of quorum: dell02 mgr: ceph08.tlocfi(active, since 58m), standbys: dell02.odtbqw, www7.rxagfn mds: 1/1 daemons up, 1 standby osd: 7 osds: 7 up (since 12h), 7 in (since 18h); 308 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 0/1 healthy, 1 recovering pools: 22 pools, 681 pgs objects: 125.10k objects, 36 GiB usage: 91 GiB used, 759 GiB / 850 GiB avail pgs: 47772/369076 objects misplaced (12.944%) 373 active+clean 308 active+clean+remapped io: client: 170 B/s rd, 0 op/s rd, 0 op/s wr [ceph: root@www7 /]# ceph health detail HEALTH_WARN 1 filesystem is degraded; 25 client(s) laggy due to laggy OSDs [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceefs is degraded [WRN] MDS_CLIENTS_LAGGY: 25 client(s) laggy due to laggy OSDs mds.ceefs.www7.drnuyi(mds.0): Client 14019719 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14124385 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14144243 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14144375 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14224103 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14224523 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14234194 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14234545 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14236841 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14237837 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14238536 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14244124 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14264236 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14266870 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14294170 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14294434 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14296012 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14304212 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14316057 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14318379 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14325518 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14328956 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14334283 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14336104 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14374237 is laggy; not evicted because some OSD(s) is/are laggy [ceph: root@www7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2.79994 root default -25 0.15999 host ceph01 1 hdd 0.15999 osd.1 up 0.15999 1.00000 -28 1.15999 host ceph03 3 hdd 0.15999 osd.3 up 0.15999 1.00000 5 hdd 1.00000 osd.5 up 1.00000 1.00000 -9 0.15999 host ceph06 2 hdd 0.15999 osd.2 up 0.15999 1.00000 -3 0.15999 host ceph07 6 hdd 0.15999 osd.6 up 0.15999 1.00000 -6 1.00000 host ceph08 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -7 0.15999 host www7 0 hdd 0.15999 osd.0 up 0.15999 1.00000 [ceph: root@www7 /]# ceph pg stat 681 pgs: 373 active+clean, 308 active+clean+remapped; 36 GiB data, 91 GiB used, 759 GiB / 850 GiB avail; 255 B/s rd, 0 op/s; 47772/369073 objects misplaced (12.944%)

1 2

18.2.4 regression: 'diskprediction_local' has failed: No module named 'sklearn'
by Harry G Coin 27 Jul '24

27 Jul '24

Upgraded to 18.2.4 yesterday. Healthy cluster reported a few minutes after the upgrade completed. Next morning, this: # ceph health detail HEALTH_ERR Module 'diskprediction_local' has failed: No module named 'sklearn' [ERR] MGR_MODULE_ERROR: Module 'diskprediction_local' has failed: No module named 'sklearn' Module 'diskprediction_local' has failed: No module named 'sklearn' Searching found this was a problem several years ago, then resolved, now returned. Any ideas? Harry Coin

2 2

Reef 18.2.4 EL8 packages ?
by Noe P. 26 Jul '24

26 Jul '24

Hi all, I find this 18.2.4 release highly confusing. Can anyone please confirm that EL8 packages are no more supported ? The small remark in the release notes qa/distros: remove centos 8 from supported distros (pr#57932, Guillaume Abrioux, Casey Bodley, Adam King, Laura Flores) is a bit disturbing, in particular between two minor releases. If yes, is the reason the missing support for EL8 distro kernels ? We could build the packages locally, but there is no v18.2.4 tag in git. Regards, Noe

3 3

Dashboard error on 18.2.4 when listing block images
by Dan O'Brien 26 Jul '24

26 Jul '24

I just yesterday updated by Ceph cluster from Quincy (17.2.7) to Reef (18.2.4, purely by accident; I didn't realize I was getting it THE EXACT SECOND it was pushed out). The upgrade resolved an issue I was having with HTTP 500 errors on the RGW UI, but seems to have created an issue in the dashboard with the Block / Images screen. I see errors that come up as alerts: 0 - Unknown Error * Http failure response for api/prometheus: 0 Unknown Error * Http failure reponse for api/prometheus/rules: 0 Unknown Error * Http failure response for api/summary: 0 Unknown Error All the other menu options on the dashboard are working. I thought it was a UI issue, but having read the notes on 18.2.4, I wonder if I've hit a bug, or if there's something wonky with the RBD images in some of our pools. We're using Ceph mostly for OpenStack. The syslog on the mgr server shows an error and traceback, when I try to use the Block / Images menu link. Notably I see Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::DiffIterate: fast diff enabled just before the crash, and there's mention of this function in the release notes. I'm going to try and use the rbd command and query some of the pools and images to see if I can narrow down specifically what's going on. If anyone's got some suggestions, I'd love to hear them. I can drop additional information from the logs if needed. I don't THINK there were errors prior to the upgrade, but I can't say for sure since I've been working in other parts of the Ceph UI. Full traceback below Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: asok(0x56360092c000) register_command rbd cache flush images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b hook 0x56361144ffc0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: asok(0x56360092c000) register_command rbd cache invalidate images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b hook 0x56361144ffc0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::ImageCtx: 0x5636176bd000: disabling zero-copy writes Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x563615cf1170 register_dispatch: dispatch_layer=3 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::cache::WriteAroundObjectDispatch: 0x56360ce5efc0 init: Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x56360bd64900 register_dispatch: dispatch_layer=1 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::SimpleSchedulerObjectDispatch: 0x5636164b9e00 SimpleSchedulerObjectDispatch: ictx=0x5636176bd000 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::SimpleSchedulerObjectDispatch: 0x5636164b9e00 init: Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x56360bd64900 register_dispatch: dispatch_layer=5 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 block_writes: 0x5636176bd000, num=1 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x563615cf1170 shut_down_dispatch: dispatch_layer=3 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 unblock_writes: 0x5636176bd000, num=0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 block_writes: 0x5636176bd000, num=1 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 unblock_writes: 0x5636176bd000, num=0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::DiffIterate: fast diff enabled Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: In function 'int librbd::api::DiffIterate<ImageCtxT>::execute() [with ImageCtxT = librbd::ImageCtx]' thread 7fd5897ac640 time 2024-07-25T13:13:15.549430+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: 341: FAILED ceph_assert(object_diff_state.size() == end_object_no - start_object_no) ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fd6f5b0104d] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fd6f5b0120b] 3: /lib64/librbd.so.1(+0x193403) [0x7fd6e5ce2403] 4: /lib64/librbd.so.1(+0x51ada7) [0x7fd6e6069da7] 5: rbd_diff_iterate2() 6: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x630bc) [0x7fd6e62f40bc] 7: /lib64/libpython3.9.so.1.0(+0x11d7a1) [0x7fd6f661e7a1] 8: PyVectorcall_Call() 9: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7fd6e62d5d50] 10: _PyObject_MakeTpCall() 11: /lib64/libpython3.9.so.1.0(+0x125133) [0x7fd6f6626133] 12: _PyEval_EvalFrameDefault() 13: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 14: _PyFunction_Vectorcall() 15: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 16: _PyEval_EvalFrameDefault() 17: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 18: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 19: _PyEval_EvalFrameDefault() 20: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 21: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 22: _PyEval_EvalFrameDefault() 23: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 24: _PyFunction_Vectorcall() 25: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 26: _PyEval_EvalFrameDefault() 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 28: _PyFunction_Vectorcall() 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 30: _PyEval_EvalFrameDefault() 31: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: *** Caught signal (Aborted) ** in thread 7fd5897ac640 thread_name:dashboard ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd6f54aa6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fd6f54f794c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fd6f5b010a7] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fd6f5b0120b] 7: /lib64/librbd.so.1(+0x193403) [0x7fd6e5ce2403] 8: /lib64/librbd.so.1(+0x51ada7) [0x7fd6e6069da7] 9: rbd_diff_iterate2() 10: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x630bc) [0x7fd6e62f40bc] 11: /lib64/libpython3.9.so.1.0(+0x11d7a1) [0x7fd6f661e7a1] 12: PyVectorcall_Call() 13: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7fd6e62d5d50] 14: _PyObject_MakeTpCall() 15: /lib64/libpython3.9.so.1.0(+0x125133) [0x7fd6f6626133] 16: _PyEval_EvalFrameDefault() 17: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 18: _PyFunction_Vectorcall() 19: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 20: _PyEval_EvalFrameDefault() 21: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 22: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 23: _PyEval_EvalFrameDefault() 24: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 25: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 26: _PyEval_EvalFrameDefault() 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 28: _PyFunction_Vectorcall() 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 30: _PyEval_EvalFrameDefault() 31: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2 7