- ceph-users - lists.ceph.io

19.2.1: HEALTH_ERR 27 osds(s) are not reachable. (Yet working normally...)
by Harry G Coin 10 Feb '25

10 Feb '25

19.2.1 complains of all osd's being unreachable, as their public address isn't in the public subnet. However, they all are within the subnet, and are working normally as well. It's embarrassing for the dashboard to glow red of a totally crippled osd roster --- while all is working normally. This existed in the previous, but was working prior to 19. Detail: Notice, for osd.0, the dashboard lists public_addr [fc00:1002:c7::44]:6807/4160993080 But, we have in the logs: 7/2/25 03:35 PM[ERR] osd.0's public address is not in 'fc00:1002:c7::/64' subnet 7/2/25 03:35 PM[ERR][ERR] OSD_UNREACHABLE: 27 osds(s) are not reachable 7/2/25 03:35 PM[ERR]Health detail: HEALTH_ERR 27 osds(s) are not reachable However, as per the osd.0 attributes, the public address for osd.0 is well inside the stated public subnet. All the osd's are similarly configured, working, and held to be unreachable at the same time, for the same reason. Tell me there's a way to fix this without waiting a further half year....

2 3

Reef: rgw daemon crashes
by Eugen Block 10 Feb '25

10 Feb '25

Hi, yesterday I upgraded a customer cluster to Reef (18.2.4). The upgrade went quite well, nothing happened for hours, until it did. One of the two RGW daemons has crashed twice in the last 12 hours. Here's one backtrace: ---snip--- Feb 06 23:19:58 storage09 conmon[2501983]: radosgw: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/redhat-linux-build/boost/include/boost/context/posix/protected_fixedsize_stack.hpp:70: boost::context::stack_context boost::context::basic_protected_fixedsize_stack<traitsT>::allocate() [with traitsT = boost::context::stack_traits]: Assertion `0 == result' failed. Feb 06 23:19:58 storage09 conmon[2501983]: *** Caught signal (Aborted) ** Feb 06 23:19:58 storage09 conmon[2501983]: in thread 7f77a10b2640 thread_name:radosgw Feb 06 23:19:58 storage09 conmon[2501983]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Feb 06 23:19:58 storage09 conmon[2501983]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f78adeba6f0] Feb 06 23:19:58 storage09 conmon[2501983]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f78adf0794c] Feb 06 23:19:58 storage09 conmon[2501983]: 3: raise() Feb 06 23:19:58 storage09 conmon[2501983]: 4: abort() Feb 06 23:19:58 storage09 conmon[2501983]: 5: /lib64/libc.so.6(+0x2871b) [0x7f78adea471b] Feb 06 23:19:58 storage09 conmon[2501983]: 6: /lib64/libc.so.6(+0x37386) [0x7f78adeb3386] Feb 06 23:19:58 storage09 conmon[2501983]: 7: /usr/bin/radosgw(+0x361cb2) [0x561ed9ef3cb2] Feb 06 23:19:58 storage09 conmon[2501983]: 8: /usr/bin/radosgw(+0x361db8) [0x561ed9ef3db8] Feb 06 23:19:58 storage09 conmon[2501983]: 9: /usr/bin/radosgw(+0x36e15e) [0x561ed9f0015e] Feb 06 23:19:58 storage09 conmon[2501983]: 10: /usr/bin/radosgw(+0x357558) [0x561ed9ee9558] Feb 06 23:19:58 storage09 conmon[2501983]: 11: /usr/bin/radosgw(+0x34546c) [0x561ed9ed746c] Feb 06 23:19:58 storage09 conmon[2501983]: 12: /usr/bin/radosgw(+0x358f0a) [0x561ed9eeaf0a] Feb 06 23:19:58 storage09 conmon[2501983]: 13: /usr/bin/radosgw(+0xb705de) [0x561eda7025de] Feb 06 23:19:58 storage09 conmon[2501983]: 14: /usr/bin/radosgw(+0x3c6aed) [0x561ed9f58aed] Feb 06 23:19:58 storage09 conmon[2501983]: 15: /lib64/libstdc++.so.6(+0xdbad4) [0x7f78ae258ad4] Feb 06 23:19:58 storage09 conmon[2501983]: 16: /lib64/libc.so.6(+0x89c02) [0x7f78adf05c02] Feb 06 23:19:58 storage09 conmon[2501983]: 17: /lib64/libc.so.6(+0x10ec40) [0x7f78adf8ac40] ---snip--- I didn't find anything helpful in the tracker, only this report on this list [0] without a response from a year ago. The other daemon on a different host seems to be stable for now. This is no multi-site deployment, just two RGWs for a single zone. Any comments/pointers are appreciated! I can file a tracker issue if this is something new. Thanks! Eugen [0] https://www.spinics.net/lists/ceph-users/msg80956.html

1 2

Ceph upgrade quincy to reef
by cristian.tavarez＠hostpapa.com 09 Feb '25

09 Feb '25

Hello I'm trying to update my ceph cluster from the latest release of quincy 17.2.8 to reef 18.2.4. After starting the upgrade with 'ceph orch upgrade start --ceph-version 18.2.4' it stopped having upgraded 3/4 of my MGR daemons and everything else stayed on quincy. Any ceph command is not working saying orchestrator module is not found and Error ENOENT: Module not found. I have already tried looking at the osd issue hinted at https://tracker.ceph.com/issues/67329 but this happened on a test cluster without any OSDs on it as well. The current status given in the upgrade is the following ceph config-key get mgr/cephadm/upgrade_state {"target_name": "quay.io/ceph/ceph:v18.2.4", "progress_id": "a9e4d538-4c98-4307-8170-a82ca8ff300c", "target_id": "2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a", "target_digests": ["quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77"], "target_version": "18.2.4", "fs_original_max_mds": null, "fs_original_allow_standby_replay": null, "error": "UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host hostname-redacted at addr (IPv4-address)", "paused": true, "daemon_types": null, "hosts": null, "services": null, "total_count": null, "remaining_count": null} How can I make the orchestrator work again to either rollback or upgrade to a newer version? the goal is to update this all the way to squid. Thank you

1 0

postgresql vs ceph, fsync
by Petr Holubec 08 Feb '25

08 Feb '25

We are evaluating pros and cons of running postgresql backed by ceph. We know that running pg on dedicated physical hw is highly recommended, but we've got our reasons. So to the question: What could happen if we switch fsync to off on postgre backed by ceph? The increase of perfomance is huge, which is the reason we are considering it. Ceph pool is running on osds connected to controllers with batteries so we could mitigate power losses. What happens when an osd run out of space? Are there other considerations?

2 1

Quick question: How to check if krbd is enabled?
by Alan Murrell 07 Feb '25

07 Feb '25

Hello, I Googled for this answer but could not find it. I am running Reef 18.2.2 . I need to find out if krbd is enabled on my cluster but I cannot figure out what command to check that. I have checked the Pool settings in the GUI but I see nothing about krbd. I suspect it is enabled, but I just need to confirm. All the posts I have found so far about enabling or disabling it have to do with doing so in Proxmox, and while we do have Proxmox connected to our cluster, the cluster was not created within Proxmox, so that option is not available in there either (nor would I expect it to be) Thanks! 😊

2 2

[RGW] Full replication gives stale recovering shard
by Gilles Mocellin 07 Feb '25

07 Feb '25

Hello Cephers, Since I was not confident in my replication status, I've done a radosgw sync init one after the other, in both of my zones. Since then, there art stale recovering shards. Incremental replication seems OK. Cluster A sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) current time 2025-02-06T19:03:10Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [65,66,77] oldest incremental change not applied: 2025-02-06T20:02:53.441838+0100 [66] 111 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101, 102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] Cluster B sync status: realm 421c9bbd-83cc-4d85-a25a-ca847f225bfe (R) zonegroup 2c5ccf52-88be-4b2f-b4b9-c55f73bbacd1 (ZG) zone aefd4003-1866-4b16-b1b3-2f308075cd1c (Z2) current time 2025-02-06T19:03:37Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 3e3b0a2b-1864-4923-93a6-237a13b51594 (Z1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 7 shards behind shards: [58,59,60,62,64,66,67] oldest incremental change not applied: 2025-02-06T20:03:27.831651+0100 [60] 81 shards are recovering recovering shards: [0,1,2,3,4,5,6,7,8,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 54,57,58,59,60,61,62,63,64,65,66,67,68,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123 ,124,126,127] Restarting full sync does nothing, except eventually adding recovering shards ! I also have some large omap objects in Z1.rgw.log pool. Where I see a lot of objects with names beginning by bucket.sync-status.*. What can I do to remove those recovering shards ? Some trimming (bilog ?), deleting objects in Z1.rgw.log pool ? -- Gilles

1 0

Cannot stop OSD
by Alan Murrell 07 Feb '25

07 Feb '25

Hello, I am running a Ceph cluster installed with cephadm. Version is 18.2.2 reef (stable). I am moving the DB/WAL from my HDDs to SSD, and have been doing fine on all the OSDs until I got to one in particular (osd.14) From the cephadm shell, when I run 'ceph orch daemon stop osd.14', nothing happens: it does not get marked as Down. If I mark is as Down in the GUI, it does show as Down then a few seconds later it gets marks as Up again. I was running 'journalctl -xf | grep osd.14' to see if any errors came up, but nothing did. Not sure where to check next to try to sort this out?

2 7

Re: RGW: HEAD ok but GET fails
by Simon Campion 07 Feb '25

07 Feb '25

We see the same symptoms. We have an S3 object for which HEAD requests return 200, but GET requests return 404. We searched the S3 gateway logs for write requests that happened on the object before GET requests started failing. Strangely, we see succeeding GET requests and then failing GET requests without any write request (such as DELETE) being logged in between. So, we're unsure whether this is related to the known bug in DELETE requests mentioned earlier in this thread (https://tracker.ceph.com/issues/63935) Excerpts from the S3 gateway logs for the object in question: ``` ... # Last upload of the file 2025-01-16 08:09:23.112 debug 2025-01-16T08:09:23.111+0000 7f4b739f7640 1 beast: 0x7f4adc1094a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.105 +0000] "POST /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?uploads=&x-id=CreateMultipartUpload HTTP/1.1" 200 359 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.006000043s 2025-01-16 08:09:23.457 debug 2025-01-16T08:09:23.456+0000 7f137c405640 1 beast: 0x7f13049b64a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.168 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=4&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.288000762s 2025-01-16 08:09:23.460 debug 2025-01-16T08:09:23.460+0000 7fc386dfa640 1 beast: 0x7fc2fef834a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.161 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=2&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.298000038s 2025-01-16 08:09:23.461 debug 2025-01-16T08:09:23.460+0000 7f4b81212640 1 beast: 0x7f4adc1094a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.158 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=5&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.302002162s 2025-01-16 08:09:23.461 debug 2025-01-16T08:09:23.460+0000 7f4b629d5640 1 beast: 0x7f4adab5e4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.162 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=3&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.298002154s 2025-01-16 08:09:23.461 debug 2025-01-16T08:09:23.460+0000 7f4ba5a5b640 1 beast: 0x7f4adaa5c4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.162 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=1&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 5242880 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.298002154s 2025-01-16 08:09:23.520 debug 2025-01-16T08:09:23.519+0000 7f4bd8ac1640 1 beast: 0x7f4adab5e4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.465 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?partNumber=6&uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=UploadPart HTTP/1.1" 200 4225533 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.053000379s 2025-01-16 08:09:23.542 debug 2025-01-16T08:09:23.541+0000 7f4ba3a57640 1 beast: 0x7f4ada9db4a0: 10.70.3.26 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:08:09:23.520 +0000] "POST /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?uploadId=2~POfVWPePjR4HDk48JZYll6Vx8izyqni&x-id=CompleteMultipartUpload HTTP/1.1" 200 548 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1 ft/s3-transfer" - latency=0.021000152s ... # GET requests return 200 2025-01-16 17:27:57.574 debug 2025-01-16T17:27:57.573+0000 7f138d427640 1 beast: 0x7f13048334a0: 10.70.3.93 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [16/Jan/2025:17:27:57.498 +0000] "GET /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=GetObject HTTP/1.1" 200 30439933 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.075000212s ... # Last write request we could find in the logs (a CopyObject request) 2025-01-17 08:11:23.721 debug 2025-01-17T08:11:23.719+0000 7fc394e16640 1 ====== starting new request req=0x7fc300cbd4a0 ===== 2025-01-17 08:11:23.721 debug 2025-01-17T08:11:23.720+0000 7fc3da6a1640 1 ====== req done req=0x7fc3009b74a0 op status=0 http_status=200 latency=0.007000001s ====== 2025-01-17 08:11:23.721 debug 2025-01-17T08:11:23.720+0000 7fc3da6a1640 1 beast: 0x7fc3009b74a0: 10.70.3.153 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:08:11:23.713 +0000] "PUT /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=CopyObject HTTP/1.1" 200 252 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.007000001s ... # Last GET request returning 200 2025-01-17 09:34:50.256 debug 2025-01-17T09:34:50.255+0000 7fc420f2e640 1 beast: 0x7fc3009b74a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:09:34:50.151 +0000] "GET /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=GetObject HTTP/1.1" 200 30439933 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.104000047s 2025-01-17 09:36:47.138 debug 2025-01-17T09:36:47.137+0000 7f13da4c1640 1 beast: 0x7f13061e64a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:09:36:47.136 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000003s 2025-01-17 10:01:57.580 debug 2025-01-17T10:01:57.579+0000 7f4bf32f6640 1 beast: 0x7f4adace14a0: 10.70.3.93 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:01:57.578 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000007s 2025-01-17 10:02:16.805 debug 2025-01-17T10:02:16.804+0000 7f4b39983640 1 beast: 0x7f4ada4504a0: 10.70.3.28 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:02:16.803 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000008s 2025-01-17 10:23:35.706 debug 2025-01-17T10:23:35.705+0000 7f142f56b640 1 beast: 0x7f13058534a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:23:35.704 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000003s 2025-01-17 10:25:38.406 debug 2025-01-17T10:25:38.405+0000 7f4b719f3640 1 beast: 0x7f4adbafd4a0: 10.70.3.28 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:25:38.404 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000007s # Now GET requests return 404s. Note there are only HEAD requests visible in the logs since the last GET request that returned a 200. 2025-01-17 10:31:56.619 debug 2025-01-17T10:31:56.617+0000 7fc37e5e9640 0 req 17300774944515556197 0.001000000s s3:get_obj iterate_obj() failed with -2 2025-01-17 10:31:56.619 debug 2025-01-17T10:31:56.618+0000 7fc35058d640 1 ====== req done req=0x7fc3011464a0 op status=-2 http_status=404 latency=0.002000001s ====== 2025-01-17 10:31:56.619 debug 2025-01-17T10:31:56.618+0000 7fc35058d640 1 beast: 0x7fc3011464a0: 10.70.3.93 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:31:56.615 +0000] "GET /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059?x-id=GetObject HTTP/1.1" 404 294 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.002000001s # HEAD requets still return 200. 2025-01-17 10:32:48.196 debug 2025-01-17T10:32:48.195+0000 7fc3a2e32640 1 beast: 0x7fc2feb7b4a0: 10.70.3.91 - obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1 [17/Jan/2025:10:32:48.194 +0000] "HEAD /buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4/blobs/sha256%3A7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 HTTP/1.1" 200 0 - "aws-sdk-go-v2/1.24.1 os/linux lang/go#1.23.4 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1" - latency=0.001000001s ``` Here's the bucket index entry. The mtime matches the time of the last CopyObject request: ``` bash-5.1$ radosgw-admin bi list --bucket=buildkit-cache-466cec97-dd8f-467d-b4bd-73df82db10c4 ... { "type": "plain", "idx": "blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059", "entry": { "name": "blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059", "instance": "", "ver": { "pool": 20, "epoch": 0 }, "locator": "", "exists": true, "meta": { "category": 1, "size": 30439933, "mtime": "2025-01-17T08:11:23.715427Z", "etag": "ea3dc44b4f3e9fa8874c0022664edeaa-6", "storage_class": "", "owner": "obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1", "owner_display_name": "obc-gitlab-runner-k8s-buildkit-cache-bucket-90b2b9dd-deab-4519-8802-082c9bc658b1", "content_type": "", "accounted_size": 30439933, "user_data": "", "appendable": false }, "tag": "26caa730-b29f-4262-ac51-6a47db5585d2.75677055.5371543061660465468", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, ``` There is a RADOS object with the relevant name, matching mtime, but with size 0: ``` bash-5.1$ rados -p general-s3.rgw.buckets.data ls | grep 7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 26caa730-b29f-4262-ac51-6a47db5585d2.75671339.1_blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 bash-5.1$ rados -p general-s3.rgw.buckets.data stat 26caa730-b29f-4262-ac51-6a47db5585d2.75671339.1_blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 general-s3.rgw.buckets.data/26caa730-b29f-4262-ac51-6a47db5585d2.75671339.1_blobs/sha256:7478e0ac0f23f94b2f27848fbcdf804a670fbf8d4bab26df842d40a10cd33059 mtime 2025-01-17T08:11:23.000000+0000, size 0 ``` We run Ceph 19.2.0 on K8s, orchestrated with Rook. We have multiple replicas for the S3 gateway. The problems started to appear after updating from Reef to Squid. So far, we only noticed the problem for a specific bucket used as a BuildKit cache. We deleted and recreated the bucket for BuildKit, and the problem appeared again after running our BuildKit jobs against the new bucket for a few days. At first, we thought the problem was caused by a bug/unwanted behavior in the CopyObject implementation. It looks like CopyObject fails to update the index but nevertheless updates the RADOS objects holding the S3 object, replacing (multiple parts of?) a large S3 object with a single object of size 0. However, the fact that we see a successful GET request ten minutes after the last CopyObject request casts doubt on the hypothesis that CopyObject is the problem. We'd be very grateful for any advice on how to troubleshoot this further. Simon

1 0

RGW issue, lost/corrupted bucket metadata/index ?
by Cyril Duval 07 Feb '25

07 Feb '25

Hello folks ! This is my first time posting here, sorry in advance if I miss some of the etiquette. I'm facing some issues with RGW and especially the radosgw-admin tool, on what appears to be lost bucket metadata, or maybe corruption of some elements of the buckets. I have opened a ticket on https://tracker.ceph.com/issues/69818 where I put more details, but in summary I have the `failure: (2) No such file or directory:` kind of error on most radosgw-admin commands interacting with these buggy buckets. Lifecycle policies seem to be affected too, as I have a `lifecycle: RGWLC::process() failed to get obj head lc.5, ret=-5` error. I'm running Ceph 18.2.2 there. Is it something familiar to anyone ? Or does anyone know where I can have a look further to better understand the issue and solve it ? I tried giving a shot at the cpp code, but unfortunately I'm no expert there, I'm lost. Thanks in advance, -- Cyril Duval cyril.duval(a)diabolocom.com

1 0

Measuring write latency (ceph osd perf)
by Jan Kasprzak 07 Feb '25

07 Feb '25

Hello, ceph users, TL;DR: how can I look into ceph cluster write latency issues? Details: we have a HDD-based cluster (with NVMe for metadata), about 20 hosts, 2 OSD per host, mostly used as RBD storage for QEMU/KVM virtual machines. From time to time our users complain about write latencies inside their VMs. I would like to be able to see when the cluster is overloaded or when the write latency is bad. What did I try so far: 1) fio inside the KVM virtual machine: fio --ioengine=libaio --direct=1 --rw=write --numjobs=1 --bs=1M --iodepth=16 --size=5G --name=/var/tmp/fio-test [...] write: IOPS=63, BW=63.3MiB/s (66.4MB/s)(5120MiB/80863msec); 0 zone resets - I am usually getting about 60 to 150 IOPS for 1MB writes 2) PostgreSQL from the KVM virtual machine, running many tiny INSERTs as separate transactions for about 10 seconds. This is where I clearly see latency spikes: Wed Dec 18 09:20:21 PM CET 2024 406.062 txn/s Wed Dec 18 09:25:21 PM CET 2024 318.974 txn/s Wed Dec 18 09:30:21 PM CET 2024 285.591 txn/s Wed Dec 18 09:35:21 PM CET 2024 191.804 txn/s Wed Dec 18 09:40:22 PM CET 2024 246.679 txn/s Wed Dec 18 09:45:22 PM CET 2024 201.005 txn/s Wed Dec 18 09:50:22 PM CET 2024 153.206 txn/s Wed Dec 18 09:55:22 PM CET 2024 124.546 txn/s Wed Dec 18 10:00:23 PM CET 2024 33.094 txn/s Wed Dec 18 10:05:23 PM CET 2024 82.659 txn/s Wed Dec 18 10:10:23 PM CET 2024 292.544 txn/s Wed Dec 18 10:15:24 PM CET 2024 453.366 txn/s The drawback of both fio and postgresql benchmark is that I am unnecessarily loading the cluster with additional work, just to measure latency. And I am not covering the whole cluster, just the OSDs on which that VM happens to have its own data. 3) ceph osd perf I don't see any single obviously overloaded OSD here, but the latencies vary nevertheless. Here are statistics computed across all OSDs from the "ceph osd perf" output: Fri Jan 3 10:12:41 AM CET 2025 average 9 9 median 5 5 3rd-q 13 13 max 70 70 Fri Jan 3 10:13:42 AM CET 2025 average 5 5 median 3 3 3rd-q 10 10 max 31 31 Fri Jan 3 10:14:42 AM CET 2025 average 3 3 median 2 2 3rd-q 4 4 max 19 19 Fri Jan 3 10:15:42 AM CET 2025 average 5 5 median 1 1 3rd-q 3 3 max 63 63 However, I am not sure what exactly these numbers actually mean - what timespan do they cover? I would like to have something like "in the last 5 minutes, 99 % of all writes committed under XXX ms". Can Ceph tell me that? What else apart from buying faster hardware can I try in order to improve the write latency for QEMU/KVM-based VMs with RBD images? Thanks for any hints. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2