ceph-users August 2023

ceph-users@ceph.io

131 participants
122 discussions

by Mark Johnson

I've been going round and round in circles trying to work this one out but I'm getting nowhere. We're running a 4 node quincy cluster (17.2.6) which recently reported the following: ceph.log-20230729.gz:2023-07-28T08:31:42.390003+0000 osd.26 (osd.26) 13834 : cluster [WRN] Large omap object found. Object: 5:6c65dd84:users.uid::callrecordings$callrecordings_rw.buckets:head PG: 5.21bba636 (5.16) Key count: 378454 Size (bytes): 75565579 This happened a week or so ago (only the key count was only just over the 200000 threshold on that occasion) and after much searching around, I found an article that suggested a deep scrub on the pg would likely resolve the issue, so I forced a deep scrub and shortly after, the warning cleared. Came into the office today to discover the above. It's on the same PG as before which is in the default.rgw.meta pool. This time, after forcing a deep-scrub on that PG, nothing changed. I did it a second time just to be sure but got the same result. I keep finding a suse article that simply suggests increasing the threshold to the previous default of 2,000,000, but other articles I read say it was lowered for a reason and that by the time it hits that figure, it's too late so I don't want to just mask it. Problem is that I don't really understand it. I found a thread here from a bit over two years ago but their issue was in the default.rgw.buckets.index pool. A step in the solution was to list out the problematic object id and check the objects per shard however, if I issue the command "rados -p default.rgw.meta ls" it returns nothing. I get a big list from "rados -p default.rgw.buckets.index ls" just nothing from the first pool. I think it may be because the meta pool isn't indexed based on something I read, but I really don't know what I'm talking about tbh. I don't know if this is helpful, but if I list out all the PGs for that pool, there are 32 PGs and 5.16 shows 80186950 bytes and 401505 keys. PG 5.c has 75298 and 384 keys. The remaining 30 PGs show zero bytes and zero keys. I'm really not sure how to troubleshoot and resolve from here. For the record, dynamic resharding is enabled in that no options have been set in the config and that is the default setting. Based on the suse article I mentioned which also references the default.rgw.meta pool, I'm gathering our issue is because we have so many buckets that are all owned by the one user and the solution is either: * delete unused buckets * create multiple users and spread buckets evenly across all users (not something we can do) * increase the threshold to stop the warning Problem is that I'm having trouble verifying this is the issue. I've tried dumping out bucket stats to a file (radosgw-admin bucket stats > bucket_stats.txt) but after three hours this is still running with no output. Thanks for your time, Mark

9 months, 1 week

veeam backup on rgw - error - op->ERRORHANDLER: err_no=-2 new_err_no=-2

by xadhoom76＠gmail.com

Hi , we have random error with rgw during a backup from veeam . Daemons go in error state. Where we can find the appropriate logs about it ? I just find out something related to this -1788> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s getting op 0 -1787> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj verifying requester -1786> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj normalizing buckets and tenants -1785> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj init permissions -1784> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj recalculating target -1783> 2023-07-31T19:51:21.169+0000 7f04567d3700 2 req 10656715914266436796 0.000000000s s3:get_obj reading permissions -1782> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 req 10656715914266436796 0.004000044s op->ERRORHANDLER: err_no=-2 new_err_no=-2 -1781> 2023-07-31T19:51:21.173+0000 7f04567d3700 2 req 10656715914266436796 0.004000044s s3:get_obj op status=0 -1780> 2023-07-31T19:51:21.173+0000 7f04567d3700 2 req 10656715914266436796 0.004000044s s3:get_obj http status=404 -1779> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 ====== req done req=0x7f03aa67a710 op status=0 http_status=404 latency=0.004000044s ====== -1778> 2023-07-31T19:51:21.173+0000 7f04567d3700 1 beast: 0x7f03aa67a710: 172.16.199.11 - veeam90 [31/Jul/2023:19:51:21.169 +0000] "GET /veeam90/Veeam/Backup/veeam90/Clients/%7Bd14cd688-57b4-4809-a1d9-14cafd191b11%7D/15fb2d73-11f5-4ef2-8d17-c9848eb40c93/Metadata/Lock/create.checkpoint/lock HTTP/1.1" 404 207 - "APN/1.0 Veeam/1.0 Backup/12.0" - latency=0.004000044s Best regards

9 months, 1 week

2024

2023

2022

2021

2020

2019

ceph-users August 2023