- ceph-users - lists.ceph.io

DB/WALL and RGW index on the same NVME
by Lukasz Borek 08 Apr '24

08 Apr '24

Hi! I'm working on a POC cluster setup dedicated to backup app writing objects via s3 (large objects, up to 1TB transferred via multipart upload process). Initial setup is 18 storage nodes (12HDDs + 1 NVME card for DB/WALL) + EC pool. Plan is to use cephadm. I'd like to follow good practice and put the RGW index pool on a no-rotation drive. Question is how to do it? - replace a few HDDs (1 per node) with a SSD (how many? 4-6-8?) - reserve space on NVME drive on each node, create lv based OSD and let rgb index use the same NVME drive as DB/WALL Thoughts? -- Lukasz

4 6

Bucket usage per storage classes
by Ondřej Kukla 08 Apr '24

08 Apr '24

Hello, I’m playing around with Storage classes in rgw and I’m looking for ways to see per bucket statistics for the diferent storage classes (for billing purposes etc.). I though that I would add another object to the bucket usage response like for multiparts - rgw.multimeta, but it’s counted under the rgw.mainIs. Is there some option to get this info? Ondrej Bucket usage I’m referring to "usage": { "rgw.main": { "size": 1333179575, "size_actual": 1333190656, "size_utilized": 1333179575, "size_kb": 1301934, "size_kb_actual": 1301944, "size_kb_utilized": 1301934, "num_objects": 4 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }

3 5

RBD Unmap busy while no "normal" process holds it.
by Nicolas FOURNIL 08 Apr '24

08 Apr '24

Hello, I've got a strange issue with ceph (ceph-adm). We use Incus (LXD reborn) and we used to add/remove containers. We had moved to Ceph for better scalability but sometimes we get this bug : https://discuss.linuxcontainers.org/t/howto-delete-container-with-ceph-rbd-… And I respawn this old bug in Incus : https://discuss.linuxcontainers.org/t/incus-0-x-and-ceph-rbd-map-is-sometim… (thanks for Stephane GRABER help !) After working on it for several hours. I find this : I do a loop who do this script (create image / map it / format it /mount it / write on it / unmount / unmap / delete image) : rbd create image1 --size 1024 --pool customers-clouds.ix-mrs2.fr.eho || exit $? RBD_DEVICE=$(rbd map customers-clouds.ix-mrs2.fr.eho/image1 || exit $? ) mkfs.ext4 ${RBD_DEVICE} || exit $? mount ${RBD_DEVICE} /media/test || exit $? dd if=/dev/zero of=/media/test/test.out sleep 10 rm /media/test/test.out || exit $? umount ${RBD_DEVICE} || exit $? rbd unmap ${RBD_DEVICE} || exit $? rbd rm customers-clouds.ix-mrs2.fr.eho/image1 || exit $? sleep 1 It works for hours without any issue BUT if I add an OSD while doing this loop I get this : + sleep 10 + rm /media/test/test.out + umount /dev/rbd0 + rbd unmap /dev/rbd0 rbd: sysfs write failed rbd: unmap failed: (16) Device or resource busy + exit 16 And ... of course the winner is always the podman of the latest OSD i've added (it holds a copy of the mounting point) root@ceph01:~# grep rbd0 /proc/*/mountinfo/proc/1415299/mountinfo:1959 1837 252:0 / /rootfs/media/test rw,relatime - ext4 /dev/rbd0 rw,stripe=16/proc/1415301/mountinfo:1959 1837 252:0 / /rootfs/media/test rw,relatime - ext4 /dev/rbd0 rw,stripe=16 root@ceph01:~# cat /proc/1415299/cmdline /run/podman-init--/usr/bin/ceph-osd-nosd.26-f--setuserceph--setgroupceph--default-log-to-file=false--default-log-to-journald=true--default-log-to-stderr=falseroot@ceph01-b15r56-2r016.ix-mrs2.fr.eho.admin:~# root@ceph01:~# cat /proc/1415301/cmdline /usr/bin/ceph-osd-nosd.26-f--setuserceph--setgroupceph--default-log-to-file=false--default-log-to-journald=true--default-log-to-stderr=falseroot@ceph01-b15r56-2r016.ix-mrs2.fr.eho.admin:~# Our setup is straight forward : latest stable Ceph release, latest Debian stable release, and deploying via ceph-adm Did someone have such a problem ? Where is the best place to report this bug ?

1 0

NFS never recovers after slow ops
by Torkil Svensgaard 06 Apr '24

06 Apr '24

Hi Cephadm Reef 18.2.1 Started draining 5 18-20 TB HDD OSDs (DB/WAL om NVMe) on one host. Even with osd_max_backfills at 1 the OSDs get slow ops from time to time which seems odd as we recently did a huge reshuffle[1] involving the same host without seeing these slow ops. I guess one difference is the disks were then only getting writes when they were added and now they are only being used for reads as they are being drained. The slow ops eventually go away but I'm seeing stuck nfsd threads from RBD exports lingering on forever. I have to reboot the NFS server to get it going again, restarting nfs-server also just hangs. Here's a stack trace from dmesg: " [Sat Apr 6 17:44:52 2024] INFO: task nfsd:52502 blocked for more than 1245 seconds. [Sat Apr 6 17:44:52 2024] Not tainted 5.14.0-362.8.1.test2.el9_3.x86_64 #1 [Sat Apr 6 17:44:52 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Apr 6 17:44:52 2024] task:nfsd state:D stack:0 pid:52502 ppid:2 flags:0x00004000 [Sat Apr 6 17:44:52 2024] Call Trace: [Sat Apr 6 17:44:52 2024] <TASK> [Sat Apr 6 17:44:52 2024] __schedule+0x20a/0x550 [Sat Apr 6 17:44:52 2024] schedule+0x2d/0x70 [Sat Apr 6 17:44:52 2024] schedule_timeout+0x11f/0x160 [Sat Apr 6 17:44:52 2024] ? xfs_trans_read_buf_map+0x133/0x300 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_btree_read_buf_block.constprop.0+0x9a/0xd0 [xfs] [Sat Apr 6 17:44:52 2024] __down_common+0x11f/0x200 [Sat Apr 6 17:44:52 2024] ? xfs_btree_read_buf_block.constprop.0+0x30/0xd0 [xfs] [Sat Apr 6 17:44:52 2024] down+0x43/0x60 [Sat Apr 6 17:44:52 2024] xfs_buf_lock+0x2d/0xe0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_find_lock+0x45/0xf0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_lookup.constprop.0+0xe4/0x170 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_get_map+0xc1/0x3a0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_buf_read_map+0x54/0x290 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_imap_to_bp+0x4e/0x70 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_imap_lookup+0x173/0x1d0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_trans_read_buf_map+0x133/0x300 [xfs] [Sat Apr 6 17:44:52 2024] ? xfs_imap_to_bp+0x4e/0x70 [xfs] [Sat Apr 6 17:44:52 2024] xfs_imap_to_bp+0x4e/0x70 [xfs] [Sat Apr 6 17:44:52 2024] xfs_iget_cache_miss+0xa2/0x370 [xfs] [Sat Apr 6 17:44:52 2024] xfs_iget+0x19f/0x270 [xfs] [Sat Apr 6 17:44:52 2024] ? __pfx_nfsd_acceptable+0x10/0x10 [nfsd] [Sat Apr 6 17:44:52 2024] xfs_nfs_get_inode.isra.0+0x5e/0xa0 [xfs] [Sat Apr 6 17:44:52 2024] xfs_fs_fh_to_dentry+0x48/0xb0 [xfs] [Sat Apr 6 17:44:52 2024] exportfs_decode_fh_raw+0x60/0x2e0 [Sat Apr 6 17:44:52 2024] ? exp_find_key+0x99/0x1e0 [nfsd] [Sat Apr 6 17:44:52 2024] ? rcu_nocb_try_bypass+0x4d/0x440 [Sat Apr 6 17:44:52 2024] ? __kmalloc+0x19b/0x370 [Sat Apr 6 17:44:52 2024] ? __pfx_put_cred_rcu+0x10/0x10 [Sat Apr 6 17:44:52 2024] ? call_rcu+0x114/0x310 [Sat Apr 6 17:44:52 2024] nfsd_set_fh_dentry+0x2b9/0x470 [nfsd] [Sat Apr 6 17:44:52 2024] fh_verify+0x1b3/0x2f0 [nfsd] [Sat Apr 6 17:44:52 2024] nfsd4_putfh+0x3e/0x70 [nfsd] [Sat Apr 6 17:44:52 2024] nfsd4_proc_compound+0x44e/0x700 [nfsd] [Sat Apr 6 17:44:52 2024] nfsd_dispatch+0x53/0x170 [nfsd] [Sat Apr 6 17:44:52 2024] svc_process_common+0x357/0x640 [sunrpc] [Sat Apr 6 17:44:52 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd] [Sat Apr 6 17:44:52 2024] ? __pfx_nfsd+0x10/0x10 [nfsd] [Sat Apr 6 17:44:52 2024] svc_process+0x12d/0x180 [sunrpc] [Sat Apr 6 17:44:52 2024] nfsd+0xd5/0x190 [nfsd] [Sat Apr 6 17:44:52 2024] kthread+0xe0/0x100 [Sat Apr 6 17:44:52 2024] ? __pfx_kthread+0x10/0x10 [Sat Apr 6 17:44:52 2024] ret_from_fork+0x2c/0x50 [Sat Apr 6 17:44:52 2024] </TASK> " Stack: " [root@cogsworth ~]# cat /proc/52502/stack [<0>] xfs_buf_lock+0x2d/0xe0 [xfs] [<0>] xfs_buf_find_lock+0x45/0xf0 [xfs] [<0>] xfs_buf_lookup.constprop.0+0xe4/0x170 [xfs] [<0>] xfs_buf_get_map+0xc1/0x3a0 [xfs] [<0>] xfs_buf_read_map+0x54/0x290 [xfs] [<0>] xfs_trans_read_buf_map+0x133/0x300 [xfs] [<0>] xfs_imap_to_bp+0x4e/0x70 [xfs] [<0>] xfs_iget_cache_miss+0xa2/0x370 [xfs] [<0>] xfs_iget+0x19f/0x270 [xfs] [<0>] xfs_nfs_get_inode.isra.0+0x5e/0xa0 [xfs] [<0>] xfs_fs_fh_to_dentry+0x48/0xb0 [xfs] [<0>] exportfs_decode_fh_raw+0x60/0x2e0 [<0>] nfsd_set_fh_dentry+0x2b9/0x470 [nfsd] [<0>] fh_verify+0x1b3/0x2f0 [nfsd] [<0>] nfsd4_putfh+0x3e/0x70 [nfsd] [<0>] nfsd4_proc_compound+0x44e/0x700 [nfsd] [<0>] nfsd_dispatch+0x53/0x170 [nfsd] [<0>] svc_process_common+0x357/0x640 [sunrpc] [<0>] svc_process+0x12d/0x180 [sunrpc] [<0>] nfsd+0xd5/0x190 [nfsd] [<0>] kthread+0xe0/0x100 [<0>] ret_from_fork+0x2c/0x50 " The nfsd treads do not recover even with nobackfill set, so the cluster is essential idle: " [root@lazy ~]# ceph -s cluster: id: XXXXXXXXXXXXXXXXXXXXXXXXX health: HEALTH_ERR nobackfill,noscrub,nodeep-scrub flag(s) set 1 scrub errors Possible data damage: 1 pg inconsistent 631 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum lazy,jolly,happy,dopey,sleepy (age 2d) mgr: jolly.tpgixt(active, since 3d), standbys: dopey.lxajvk, lazy.xuhetq mds: 1/1 daemons up, 2 standby osd: 537 osds: 537 up (since 8h), 537 in (since 10d); 917 remapped pgs flags nobackfill,noscrub,nodeep-scrub data: volumes: 1/1 healthy pools: 15 pools, 10849 pgs objects: 548.99M objects, 1.1 PiB usage: 1.9 PiB used, 2.3 PiB / 4.2 PiB avail pgs: 97810419/3182590113 objects misplaced (3.073%) 9931 active+clean 893 active+remapped+backfill_wait 24 active+remapped+backfilling 1 active+clean+inconsistent io: client: 3.5 KiB/s rd, 2.0 MiB/s wr, 5 op/s rd, 115 op/s wr " Any ideas on how to get the nfsd threads to recover? There must be stuck ceph I/O somewhere which never time out or something like that? Mvh. Torkil [1] https://www.spinics.net/lists/ceph-users/msg81549.html -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

2 2

question regarding access cephFS from external network.
by elite_stu＠163.com 06 Apr '24

06 Apr '24

Dears , I have a question about cephfs port 6789 is with type of ClusterIP, then how can i access it from external network? [root@vm-01 examples]# kubectl get svc -nrook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rook-ceph-mgr ClusterIP 10.106.49.230 <none> 9283/TCP 6d23h rook-ceph-mgr-dashboard ClusterIP 10.96.37.100 <none> 8443/TCP 6d23h rook-ceph-mgr-dashboard-external-https NodePort 10.108.78.191 <none> 8443:31082/TCP 6d21h rook-ceph-mon-a ClusterIP 10.107.62.113 <none> 6789/TCP,3300/TCP 3h12m rook-ceph-mon-b ClusterIP 10.103.94.71 <none> 6789/TCP,3300/TCP 3h12m rook-ceph-mon-d ClusterIP 10.110.210.113 <none> 6789/TCP,3300/TCP 3h1m rook-ceph-rgw-my-store ClusterIP 10.98.150.97 <none> 80/TCP 4d6h rook-ceph-rgw-my-store-external NodePort 10.111.249.203 <none> 80:30514/TCP 4d5h

1 0

cephadm: daemon osd.x on yyy is in error state
by Zakhar Kirpichenko 06 Apr '24

06 Apr '24

Hi, A disk failed in our cephadm-managed 16.2.15 cluster, the affected OSD is down, out and stopped with cephadm, I also removed the failed drive from the host's service definition. The cluster has finished recovering but the following warning persists: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.11 on ceph02 is in error state Is it possible to remove or suppress this warning without having to completely remove the OSD? I would appreciate any advice or pointers. Best regards, Zakhar

2 3

"ceph orch daemon add osd" deploys broken OSD
by service.plant＠ya.ru 06 Apr '24

06 Apr '24

Hi everybody. I've faced the situation when I cannot redeploy OSD on a new disk So, I need to replace osd.30 cuz disk always reports about problems with I\O. I do `ceph orch daemon osd.30 --replace` Then I zap DB ``` root@server-2:/# ceph-volume lvm zap /dev/ceph-db/db-88 --> Zapping: /dev/ceph-db/db-88 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-db/db-88 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0247342 s, 424 MB/s --> Zapping successful for: <LV: /dev/ceph-db/db-88> ``` And now zap DATA ``` root@server-2:/# ceph-volume lvm zap /dev/sdn --> Zapping: /dev/sdn --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdn bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.35239 s, 7.8 MB/s --> Zapping successful for: <Raw Device: /dev/sdn> ``` Okay, now disk is ready and orchestrator confirms it ``` root@server-1:~# ceph orch device ls host server-2 --refresh server-2 /dev/sdn hdd ST18000NM008J_5000c500d80398bf 16.3T Yes 4m ago ``` Now its time for orchestrator to add new osd ``` root@server-1:~# ceph orch daemon add osd server-2:data_devices=/dev/sdn,db_devices=/dev/ceph-db/db-88 Created no osd(s) on host server-2; already created? ``` But it gives osd.30 in state down. If I try to run systemd service manually, it cannot start because ``` Apr 02 12:30:41 server-2 systemd[1]: Started Ceph osd.30 for ea98e312-dfd9-11ee-a226-33f018c3a407. Apr 02 12:30:41 server-2 bash[3316003]: /bin/bash: /var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/unit.run: No such file or directory Apr 02 12:30:41 server-2 systemd[1]: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407(a)osd.30.service: Main process exited, code=exited, status=127/n/a Apr 02 12:30:41 server-2 bash[3316014]: /bin/bash: /var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/unit.poststop: No such file or directory Apr 02 12:30:41 server-2 systemd[1]: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407(a)osd.30.service: Failed with result 'exit-code'. Apr 02 12:30:51 server-2 systemd[1]: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407(a)osd.30.service: Scheduled restart job, restart counter is at 1. Apr 02 12:30:51 server-2 systemd[1]: Stopped Ceph osd.30 for ea98e312-dfd9-11ee-a226-33f018c3a407. ``` And even if I try to redeploy osd.30 by ceph orch osd redeploy osd.30 then I get the error in ceph -W cephadm ``` 2024-04-02T12:41:39.856767+0000 mgr.server-2.opelxj (mgr.2994187) 5453 : cephadm [INF] Reconfiguring daemon osd.30 on server-2 2024-04-02T12:41:41.048352+0000 mgr.server-2.opelxj (mgr.2994187) 5454 : cephadm [ERR] cephadm exited with an error code: 1, stderr: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd-30 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd-30 Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd.30 /usr/bin/docker: stdout /usr/bin/docker: stderr Error response from daemon: No such container: ceph-ea98e312-dfd9-11ee-a226-33f018c3a407-osd.30 Reconfig daemon osd.30 ... Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 10700, in <module> File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 10688, in main File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 6620, in command_deploy_from File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 6638, in _common_deploy File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 6666, in _dispatch_deploy File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 3792, in deploy_daemon File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 3078, in create_daemon_dirs File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/cephadm.8c89112927b45a1984d03fb02785df709234bdb856619c217e1ad5d54aebef2b/__main__.py", line 708, in write_new IsADirectoryError: [Errno 21] Is a directory: '/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/config.new' -> '/var/lib/ceph/ea98e312-dfd9-11ee-a226-33f018c3a407/osd.30/config' ``` Could anyone point direction where I should dig into? Thanks in advance!

2 2

How to Identify Bottlenecks in RBD job
by Henry lol 05 Apr '24

05 Apr '24

Hello, I executed a long-running rbd flatten job on the client and expected one of the resources, such as CPU, memory, or network I/O, to reach its maximum usage. However, they didn’t. How can I find the bottleneck to improve performance? Specifically, cpu usage of each msgr-worker thread of rbd is under 60%, network usage is under 30% and memory is rarely used.

1 1

Upgraded to Quincy 17.2.7: some S3 buckets inaccessible
by Lorenz Bausch 05 Apr '24

05 Apr '24

Hi everybody, we upgraded our containerized Red Hat Pacific cluster to the latest Quincy release (Community Edition). The upgrade itself went fine, the cluster is HEALTH_OK, all daemons run the upgraded version: ---- %< ---- $ ceph -s cluster: id: 68675a58-cf09-4ebd-949c-b9fcc4f2264e health: HEALTH_OK services: mon: 5 daemons, quorum node02,node03,node04,node05,node01 (age 25h) mgr: node03.ztlair(active, since 25h), standbys: node01.koymku, node04.uvxgvp, node02.znqnhg, node05.iifmpc osd: 408 osds: 408 up (since 22h), 408 in (since 7d) rgw: 19 daemons active (19 hosts, 1 zones) data: pools: 11 pools, 8481 pgs objects: 236.99M objects, 544 TiB usage: 1.6 PiB used, 838 TiB / 2.4 PiB avail pgs: 8385 active+clean 79 active+clean+scrubbing+deep 17 active+clean+scrubbing io: client: 42 MiB/s rd, 439 MiB/s wr, 2.15k op/s rd, 1.64k op/s wr --- $ ceph versions | jq .overall { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 437 } ---- >% ---- After all the daemons were upgraded we started noticing some RGW buckets which are inaccessible. s3cmd failed with NoSuchKey: ---- %< ---- $ s3cmd la -l ERROR: S3 error: 404 (NoSuchKey) ---- >% ---- The buckets still exists according to "radosgw-admin bucket list". Out of the ~600 buckets, 13 buckets are unaccessible at the moment: ---- %< ---- $ radosgw-admin bucket radoslist --tenant xy --uid xy --bucket xy 2024-04-03T12:13:40.607+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed 2024-04-03T12:13:40.609+0200 7f0dbf4c4680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for xy:xy[6955f50e-5b23-4534-9b77-c7078f60f0d0.171713434.3]) failed ---- >% ---- The affected buckets are comparatively large, around 4 - 7 TB, but not all buckets of that size are affected. Using "rados -p rgw.buckets.data ls" it seems like all the objects are still there, although "rados -p rgw.buckets.data get objectname -" only prints unusable (?) binary data, even for objects of intact buckets. Overall we're facing around 60 TB of customer data which are just gone at the moment. Is there a way to recover from this situation or further narrowing down the root cause of the problem? Kind regards, Lorenz

3 6

Re: Dashboard and Object Gateway
by Lawson, Nathan 04 Apr '24

04 Apr '24

I am running into this issue as well with a cephadm reef deploy, are there any updates?

1 0