- ceph-users - lists.ceph.io

rgw gateways query via /api/rgw/daemon
by Szabo, Istvan (Agoda) 21 Feb '25

21 Feb '25

Hi, I'd like to query the gateways via the rgw daemon api, but I got this error: [src/ceph_client.rs:97:9] s.clone() = "{\"status\": \"500 Internal Server Error\", \"detail\": \"The server encountered an unexpected condition which prevented it from fulfilling the request.\", \"request_id\": \"089f9e0a-3bce-4902-91ee-cdf9a4f57839\"} " [src/ceph_client.rs:193:13] l = Err( "expected array value", ) Checked couple of threads pointing to the rgw_dns_name setting in ceph.conf (didn't work) and also in the mailist there is a thread but without solution. Also went through this: https://docs.ceph.com/en/quincy/mgr/dashboard/#enabling-the-object-gateway-… but didn't help. On the dashboard I set our certificates, in the ceph logs I can see these: 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.186, port=8085, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8081, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8082, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8085, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.187, port=8080, ssl=False 2025-02-21T09:51:49.394+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.199.84, port=8080, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8082, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8085, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8081, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8080, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=x.x.207.185, port=8083, ssl=False 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 258, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 190, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 108, in list 'port': int(metadata['frontend_config#0'].split('port=')[1]) IndexError: list index out of range 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard ERROR request] [10.105.176.214:49981] [GET] [500] [0.011s] [admin] [513.0B] /api/rgw/daemon 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "1739b2ab-5450-4c16-b934-82bf67198cbc"} '] 2025-02-21T09:51:49.398+0700 7f5310dd4640 0 [dashboard INFO request] [10.105.176.214:49981] [GET] [500] [0.011s] [admin] [513.0B] /api/rgw/daemon 2025-02-21T09:51:49.474+0700 7f530ddce640 0 [dashboard INFO request] [10.105.176.214:49983] [GET] [200] [0.010s] [admin] [79.0B] /api/feature_toggles 2025-02-21T09:51:49.478+0700 7f530e5cf640 0 [dashboard INFO request] [10.105.176.214:49968] [GET] [200] [0.013s] [admin] [1.6K] /api/summary My ceph version is: 17.2.7 non containerized. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

ATTN: DOCS /api/cluster/user/export
by Kalló Attila 21 Feb '25

21 Feb '25

Hi, Could you please change this doc https://docs.ceph.com/en/quincy/mgr/ceph_api/#post--api-cluster-user-export as it doesn't accept simple strings, but a json array. i.e. I think this: { > "entities": "string" > } should be replaced by something like: > > { > "entities": ["string"] > } Here is an example cURL query just in case: > > ~$ curl -kL -H "Content-Type: application/json" -H "Accept: application/vnd.ceph.api.v1.0+json" -H "Authorization: Bearer $TOKEN" "${URL}api/cluster/user/export" -X POST -d '{ "entities": ["client.admin"] }' |jq -r . > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 206 100 174 100 32 6960 1280 --:--:-- --:--:-- --:--:-- 9809 > [client.admin] > key = ****KEY**** > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" Regards, Attila

4 4

RGW Squid radosgw-admin lc process not working
by Adam Prycki 20 Feb '25

20 Feb '25

Hello, this weekend I was trying to test if squid still suffers from orphaned objects from multipart uploads. https://tracker.ceph.com/issues/44660 I've previously encountered major issues with orphaned objects when using lifecycle transitions and server side copies. Especially on slower clusters. So far I couldn't induce orphaned objects during upload and server side copy :D I couldn't quickly test lifecycle transition because I cannot get `radosgw-admin lc process` to work on any bucket. I'm using 19.2.0 cephadm cluster with fresh rgw pools (I've removed all rgw and created a fresh instnace) Lifecycle config is very simple and based on aws example: <LifecycleConfiguration> <Rule> <ID>transition</ID> <Filter> <Prefix></Prefix> </Filter> <Status>Enabled</Status> <Transition> <Days>0</Days> <StorageClass>STANDARD_IA</StorageClass> </Transition> </Rule> </LifecycleConfiguration> I have STANDARD and STANDARD_IA storage classes in my placement. I'm testing on a bucket with 2 objects, one on STANDARD and one on STANDARD_IA. [ { "key": "default-placement", "val": { "index_pool": "n8.rgw.buckets.default-placement.index", "storage_classes": { "STANDARD": { "data_pool": "n8.rgw.buckets.default-placement.data.STANDARD" }, "STANDARD_IA": { "data_pool": "n8.rgw.buckets.default-placement.data.STANDARD_IA" } }, "data_extra_pool": "n8.rgw.buckets.default-placement.non-ec", "index_type": 0, "inline_data": true } } ] running `radosgw-admin lc process` results in this error lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.14 running `radosgw-admin lc process --bucket test1` results in no output ceph claims that lifecycle executed but no data was copied between pools radosgw-admin lc list [ { "bucket": ":test1:a0f7b0da-04a7-4053-b710-a15897233e86.334949.1", "shard": "lc.14", "started": "Sun, 02 Feb 2025 22:45:37 GMT", "status": "COMPLETE" } ] I've looked on ceph issue tracker but couldn't find anyone mentioning similar lifecycle issues on Squid. (My test buckets are not versioned.) Is this a bug or some kind of misconfiguration? Best regards Adam Prycki

2 2

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

Broke libvirt on compute node due to Ceph Luminous to Nautilus Upgrade
by Pardhiv Karri 19 Feb '25

19 Feb '25

Hi, We recently upgraded our Ceph from Luminous to Nautilus and upgraded the ceph clients on OpenStack (using rbd). All went well and after a few days, we randomly saw instances getting stuck with libvirt_qemu_exporter, which is getting the libvirt stuck on Openstack compute nodes. We had to kill those instances process, and then libvirt is returning. But the issue is happening again on the compute nodes with other instances. Upon doing some research, I found that we need to migrate the instances to use the latest (nautilus) ceph client, as they still use the old(luminous) client when spun up. The only way to get them to have the Nautilus client is to live migrate or reboot. We have thousands of instances, and doing any of those takes a long time without impacting the customer. Is there any other fix to solve this issue without migrating or rebooting the instances? Error on compute hosts: (renamed host and instance id) Feb 18 00:08:00 cmp03 libvirtd[5362]: 2025-02-18 00:08:00.510+0000: 5627: warning : qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for domain instance-009141b8; current job is (query, none) owned by (5628 remoteDispatchDomainBlockStats, 0 <null>) for (322330s, 0s) Thanks, Pardhiv

2 5

Ceph calculator
by Devender Singh 18 Feb '25

18 Feb '25

Hello all Do we have a good cluster design calculator which can suggest failure domain and pool size and min size according the number of nodes and drive and their size for both replicated and EC pools. Regards Dev

3 2

"event": "header_read"
by Szabo, Istvan (Agoda) 18 Feb '25

18 Feb '25

Hi, I haven't really see much about this kind of event in ceph KB: "event": "header_read", "time": "2025-02-18T14:20:00.610389+0700", "duration": 4294967295.999999 However we have huge latencies caused by this operation: "description": "osd_repop(client.3694832962.0:140533664 26.561 e748587/706771 26:86be7203:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.3654639937.1.43:head v 748587'789007122, mlcod=748587'789007120)", "initiated_at": "2025-02-18T14:20:00.610390+0700", "age": 87.067761466999997, "duration": 0.035560873999999999, "type_data": { "flag_point": "commit sent; apply or cleanup", "events": [ { "event": "initiated", "time": "2025-02-18T14:20:00.610390+0700", "duration": 0 }, { "event": "throttled", "time": "2025-02-18T14:20:00.610390+0700", "duration": 0 }, { "event": "header_read", "time": "2025-02-18T14:20:00.610389+0700", "duration": 4294967295.999999 }, { "event": "all_read", "time": "2025-02-18T14:20:00.610402+0700", "duration": 1.3264e-05 }, { "event": "dispatched", "time": "2025-02-18T14:20:00.610405+0700", "duration": 2.3549999999999999e-06 }, { "event": "queued_for_pg", "time": "2025-02-18T14:20:00.610409+0700", "duration": 3.8770000000000003e-06 }, { "event": "reached_pg", "time": "2025-02-18T14:20:00.610429+0700", "duration": 2.0027e-05 }, { "event": "started", "time": "2025-02-18T14:20:00.610438+0700", "duration": 9.8290000000000003e-06 }, { "event": "commit_sent", "time": "2025-02-18T14:20:00.645931+0700", "duration": 0.035492616999999997 }, { "event": "done", "time": "2025-02-18T14:20:00.645951+0700", "duration": 2.0007e-05 } Is there a way to track it down what does this mean? Thanks ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

How to reduce CephFS num_strays effectively?
by jinfeng.biao＠cba.com.au 18 Feb '25

18 Feb '25

Hello folks, We had an issue with the num_strays hit 1 million recently. As a workaround, max bal was increased to 125,000. The stray_num keeps growing at 25k per day. After a recent observation of 10TiB file deletion, the relevant application was stopped. Then we increased purging options to below values mds advanced filer_max_purge_ops 40 mds advanced mds_max_purge_files 1024 mds advanced mds_max_purge_ops 32768 mds advanced mds_max_purge_ops_per_pg 3 And run "du -hsx" to the top level directory mounted to the app that does massive deletion. Despite all above, strays still growing at 60K per day. There are a lot more applications using this CephFS filesystem and only this app is observed perform deletion at this scale. I'm wondering what would be the effective way to cleanup the strays in this situation while making the least impact to production. Note: We are on 14.2.6 thanks James Biao

5 4

Dashboard soft freeze with 19.2.1
by Nicola Mori 18 Feb '25

18 Feb '25

Dear Ceph users, after the upgrade to 19.2.1 I experience a soft freeze of the web UI (old version, activated with `ceph dashboard feature disable dashboard`). When I click on the "Cluster status" widget I get only a blank small tooltip window, instead of e.g. the list of current warnings. After this all the other widgets and menu entries become unresponsive until I refresh the page. Is this a known/common issue? Thanks, Nicola

2 1