[RGW] Space usage vastly overestimated since Octopus upgrade
Hi, Since upgrading from Nautilus 14.2.9 -> Octopus 15.2.3 two weeks ago we are seeing large upticks in the reported size (both space and object count) for a number of our RGW users. It does not seem to be isolated to just one user, so I don't think it's something wrong in the users' usage patterns. Users are hitting their quotas very quickly even though they are not writing anywhere near the reported space usage. Has anyone else seen this happen to them? I'm not sure what the most useful debugging information I could send would be. For example, here is a bucket that all of a sudden reports that it has 18446744073709551615 objects! The actual count should be around 20,000. [root@objproxy01 ~]# radosgw-admin bucket stats --bucket=droot-2020 { "bucket": "droot-2020", "num_shards": 32, "tenant": "", "zonegroup": "29946069-33ce-49b7-b93d-de8c95a0c344", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "8b980d5b-23de-41f9-8b14-84a5bbc3f1c9.93433056.64", "marker": "8b980d5b-23de-41f9-8b14-84a5bbc3f1c9.93433056.64", "index_type": "Normal", "owner": "-droot", "ver": "0#12052,1#15700,2#11033,3#11079,4#11521,5#13708,6#12427,7#10442,8#12769,9#11965,10#12820,11#11015,12#12073,13#11741,14#11851,15#124 97,16#10611,17#11652,18#10162,19#13699,20#9519,21#14224,22#13575,23#12635,24#9413,25#11450,26#12700,27#13122,28#10762,29#14674,30#10809,31#1223 2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0 ,27#0,28#0,29#0,30#0,31#0", "mtime": "2020-06-29T15:14:49.363664Z", "creation_time": "2020-02-04T20:36:40.752748Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551615 }, "rgw.main": { "size": 11612169555286, "size_actual": 11612211085312, "size_utilized": 11612169555286, "size_kb": 11340009332, "size_kb_actual": 11340049888, "size_kb_utilized": 11340009332, "num_objects": 20034 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } The user who owns that bucket above is reportedly using 1.3PB of space, but the known usage is 1/10th I would guess of that until we did the upgrade. [root@objproxy01 ~]# radosgw-admin user stats --uid=-droot { "stats": { "size": 1428764900976977, "size_actual": 1428770491326464, "size_utilized": 0, "size_kb": 1395278223611, "size_kb_actual": 1395283682936, "size_kb_utilized": 0, "num_objects": 2604800 }, "last_stats_sync": "2020-06-29T13:42:26.474035Z", "last_stats_update": "2020-06-29T13:42:26.471413Z" } This seems to be happening with may users who actively write data in our Object Store. Any help appreciated! Thanks, Liam University of Maryland Institute for Advanced Computer Studies
Den mån 29 juni 2020 kl 17:27 skrev Liam Monahan <liam@umiacs.umd.edu>:
For example, here is a bucket that all of a sudden reports that it has 18446744073709551615 objects! The actual count should be around 20,000.
"rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551615 },
That number is a small negative 64bit signed value, printed as an unsigned 64bit integer. Seems like the counter underflowed. 2^64 = 18446744073709551616 -- May the most significant bit of your life be positive.
You can ignore rgw.none details, it dont make sense today from our experience Still dont know why dev dont cleanup bucket with those rgw.none stats... Some of our buckets got it others new ones no. -----Mensaje original----- De: Janne Johansson <icepic.dz@gmail.com> Enviado el: martes, 30 de junio de 2020 8:40 Para: Liam Monahan <liam@umiacs.umd.edu> CC: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Re: [RGW] Space usage vastly overestimated since Octopus upgrade Den mån 29 juni 2020 kl 17:27 skrev Liam Monahan <liam@umiacs.umd.edu>:
For example, here is a bucket that all of a sudden reports that it has 18446744073709551615 objects! The actual count should be around 20,000.
"rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551615 },
That number is a small negative 64bit signed value, printed as an unsigned 64bit integer. Seems like the counter underflowed. 2^64 = 18446744073709551616 -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Reposting here since it seemed to start a new thread. Thanks, both. That’s a useful observation. I wonder what I can try to get accurate user stats. All of our users are quota-ed, so wrong users stats actually stop them from writing data. Since stats are only updated on write: I have some users who are inactive and their stats are correct. I have other users who have been actively writing. I see users who have up to 55 times the expected vs. actual size. I looped over buckets manually via the Admin Ops API and pulled the stats for all of the user’s buckets and summed these and compared that to the output from “radosgw-admin user stats" I would guess that underflowing counters could be one explanation, but there may be other things going wrong in the stats aggregation... Thanks, Liam
I downgraded a single RGW node in the cluster back to Nautilus (14.2.10), ran "radosgw-admin user stats --uid=<username> --sync-stats" and the usage calculated to the correct usage values. It seems like there is a potential bug in how Octopus is calculating its user stats. ~ Liam
participants (3)
-
EDH - Manuel Rios
-
Janne Johansson
-
Liam Monahan