- ceph-users - lists.ceph.io

Rados Gateway sync requests are not balance between nodes
by Nghia Viet Tran 15 Jul '20

15 Jul '20

Hello everyone, Below is our current setup, the master zone is on the master datacenter and the backup zone is on the standby datacenter which has very slow bandwidth internet connection. The client application connects to the Master zone load balancer for uploading objects to cluster. Later on, the Gate way nodes from the Backup zone will synchronize new objects into the Backup cluster by requesting object data to the load balancer of the master zone. [A screenshot of a cell phone Description automatically generated] The current connection inside Gateway nodes of the backup zone [A screenshot of a computer Description automatically generated] From the above image, you can see that the first Gateway node create a lot of connection to the master zone balancer (cephmm-03) for checking objects status. That means this gateway mostly will take the role for downloading new objects from the master zone to the backup zone (very high load on this node). On the other hand, the second Gateway node just has only 2 connections to the master node balancer (mostly free all the time). This second node only increase connections if the first node (or may be the third node) is down… Can you explain this behavior of Ceph in this case and how to let all the gateway node in active mode? I appreciate any comments from you! -- Nghia Viet Tran (Mr)

1 0

Ceph and Red Hat Summit 2020
by Bobby 15 Jul '20

15 Jul '20

Hi, Any Ceph related event in Red Hat Summit 2020 happening today? BR Bobby !

1 0

Re: mon_osd_down_out_subtree_limit not working?
by Frank Schilder 15 Jul '20

15 Jul '20

Yes. After the time-out of 600 secs the OSDs got marked down, all PGs got remapped and recovery/rebalancing started as usual. In the past, I did service on servers with the flag noout set and would expect that mon_osd_down_out_subtree_limit=host has the same effect when shutting down an entire host. Unfortunately, in my case these two settings behave differently. If I understand the documentation correctly, the OSDs should not get marked out automatically. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 14 July 2020 04:32:05 To: Frank Schilder Subject: Re: [ceph-users] mon_osd_down_out_subtree_limit not working? Did it start rebalancing? > On Jul 13, 2020, at 4:29 AM, Frank Schilder <frans(a)dtu.dk> wrote: > > if I shut down all OSDs on this host, these OSDs should not be marked out automatically after mon_osd_down_out_interval(=600) seconds. I did a test today and, unfortunately, the OSDs do get marked as out. Ceph status was showing 1 host down as expected.

2 7

Re: Poor Windows performance on ceph RBD.
by Frank Schilder 15 Jul '20

15 Jul '20

> If I may ask, which version of the virtio drivers do you use? https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-vir… Looks like virtio-win-0.1.185.* > And do you use caching on libvirt driver level? In the ONE interface, we use DISK = [ driver = "raw" , cache = "none"] which translates to <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> in the XML. We have no qemu settings in the ceph.conf. Looks like caching is disabled. Not sure if this is the recommended way though and why caching is disabled by default. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: André Gemünd <andre.gemuend(a)scai.fraunhofer.de> Sent: 13 July 2020 11:18 To: Frank Schilder Subject: Re: [ceph-users] Re: Poor Windows performance on ceph RBD. If I may ask, which version of the virtio drivers do you use? And do you use caching on libvirt driver level? Greetings André ----- Am 13. Jul 2020 um 10:43 schrieb Frank Schilder frans(a)dtu.dk: >> > To anyone who is following this thread, we found a possible explanation for >> > (some of) our observations. > >> If someone is following this, they probably want the possible >> explanation and not the knowledge of you having the possible >> explanation. > >> So you are saying if you do eg. a core installation (without gui) of >> 2016/2019 disable all services. The fio test results are signficantly >> different to eg. a centos 7 vm doing the same fio test? Are you sure >> this is not related to other processes writing to disk? > > Right, its not an explanation but rather a further observation. We don't really > have an explanation yet. > > Its an identical installation of both server versions, same services configured. > Our operators are not really into debugging Windows, that's why we were asking > here. Their hypothesis is, that the VD driver for accessing RBD images has > problems with Windows servers newer than 2016. I'm not a Windows guy, so can't > really comment on this. > > The test we do is a simple copy-test of a single 10g file and we monitor the > transfer speed. This info was cut out of this e-mail, the original report for > reference is: > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/ANHJQZLJT4… > . > > We are very sure that it is not related to other processes writing to disk, we > monitor that too. There is also no competition on the RBD pool at the time of > testing. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Sent: 13 July 2020 10:24 > To: ceph-users; Frank Schilder > Subject: RE: [ceph-users] Re: Poor Windows performance on ceph RBD. > >>> To anyone who is following this thread, we found a possible > explanation for >>> (some of) our observations. > > If someone is following this, they probably want the possible > explanation and not the knowledge of you having the possible > explanation. > > So you are saying if you do eg. a core installation (without gui) of > 2016/2019 disable all services. The fio test results are signficantly > different to eg. a centos 7 vm doing the same fio test? Are you sure > this is not related to other processes writing to disk? > > > > -----Original Message----- > From: Frank Schilder [mailto:frans@dtu.dk] > Sent: maandag 13 juli 2020 9:28 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Re: Poor Windows performance on ceph RBD. > > To anyone who is following this thread, we found a possible explanation > for (some of) our observations. > > We are running Windows servers version 2016 and 2019 as storage servers > exporting data on an rbd image/disk. We recently found that Windows > server 2016 runs fine. It is still not as fast as Linux + SAMBA share on > an rbd image (ca. 50%), but runs with a reasonable sustained bandwidth. > With Windows server 2019, however, we observe near-complete stall of > file transfers and time-outs using standard copy tools (robocopy). We > don't have an explanation yet and are downgrading Windows servers where > possible. > > If anyone has a hint what we can do, please let us know. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Dipl.-Inf. André Gemünd, Leiter IT / Head of IT Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemuend(a)scai.fraunhofer.de Tel: +49 2241 14-2193 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

1 1

how to configure cephfs-shell correctly
by Zhenshi Zhou 15 Jul '20

15 Jul '20

Hi all, I want to use cephfs-shell dealing with operations like directory creation, instead of mounting the root directory and create manually. But I get errors when I execute the command 'cephfs-shell'. Traceback (most recent call last): File "./cephfs-shell", line 9, in <module> import cephfs as libcephfs ModuleNotFoundError: No module named 'cephfs' CentOS7 use python2 and I've already installed python3 in the system. What else should I do, reinstall libcephfs? Thanks

2 9

YUM doesn't find older release version of nautilus
by Lee, H. (Hurng-Chun) 15 Jul '20

15 Jul '20

Hi, We use the official Ceph RPM repository (http://download.ceph.com/rpm- nautilus/el7) for installing packages on the client nodes running CentOS7. But we noticed today that the repo only provides the latest version (2:14.2.10-0.el7) of nautilus so that we couldn't install an older (2:14.2.7-0.el7) version of the ceph-common package. The ceph.repo file contains: --- [Ceph] name=Ceph packages for $basearch baseurl=http://download.ceph.com/rpm-nautilus/el7/$basearch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc --- and --- $ yum clean all $ yum makecache $ yum list --disablerepo=* --enablerepo=Ceph --showduplicates ceph- common Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Available Packages ceph-common.x86_64 2:14.2.10- 0.el7 Ceph --- Does it mean that the official repo no longer provides RPM packages for older versions? Thanks! Cheers, Hong -- Hurng-Chun (Hong) Lee, PhD ICT manager Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging Radboud University Nijmegen e-mail: h.lee(a)donders.ru.nl tel: +31(0) 243610977 web: http://www.ru.nl/donders/

3 3

Web UI errors
by Will Payne 14 Jul '20

14 Jul '20

Hi, I’ve been getting errors in the NFS section of the web interface. I’ve just tried upgrading to 15.2.4 to see if that helped but no joy. The initial NFS page loads OK and when I click Add, a form loads. However, when this form attempts to update its values, I get a red box informing me that the server returned an error 500. it makes four HTTP calls - to daemon, clients, filesystems and fsals - these all fail. Any suggestions on what might be wrong or where some useful logs might be? Ta, Will

2 1

User stats - Object count wrong in Octopus?
by David Monschein 14 Jul '20

14 Jul '20

Hi All, Sorry for the double email, I accidentally sent the previous e-mail with an accidental KB shortcut before it was finished :) I'm investigating what appears to be a bug in RGW stats. This is a brand new cluster running 15.2.3 One of our customers reached out, saying they were hitting their quota (S3 error: 403 (QuotaExceeded)). The user-wide max_objects quota we set is 50 million objects, so this would be impossible since the entire cluster isn't even close to 50 million objects yet: [root@os1 ~]# ceph status | grep objects objects: 7.58M objects, 6.8 TiB The customer in question has three buckets, and if I query the bucket stats, the total number of objects for all 3 buckets comes to about ~372k: [root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 | grep num_objects "num_objects": 324880 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo | grep num_objects "num_objects": 47476 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-test | grep num_objects "num_objects": 1 But things get interesting when I query the user stats: [root@os1 ~]# radosgw-admin user stats --uid=user-in-question | grep num_objects "num_objects": 52543794 How is Ceph arriving at 52+ million objects? Here is the full output from the bucket stats and user stats if it's any help. Thanks for any assistance. [root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 { "bucket": "df-fs1", "num_shards": 491, "tenant": "", "zonegroup": "a1b72aa0-eb06-4a96-8af4-db39d4dc3e09", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9736264.34", "marker": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9736264.21", "index_type": "Normal", "owner": "01c235fd-fc9b-4c54-a2a5-b38054608b9e", "ver": "0#57,1#58,2#59,3#56,4#63,5#62,6#52,7#61,8#48,9#58,10#56,11#58,12#64,13#45,14#57,15#66,16#66,17#55,18#52,19#60,20#74,21#59,22#57,23#47,24#60,25#52,26#62,27#58,28#60,29#47,30#57,31#45,32#48,33#56,34#64,35#68,36#47,37#49,38#62,39#48,40#64,41#44,42#60,43#60,44#60,45#59,46#65,47#47,48#57,49#57,50#63,51#68,52#61,53#57,54#46,55#59,56#57,57#41,58#59,59#58,60#57,61#53,62#52,63#45,64#55,65#58,66#54,67#43,68#60,69#51,70#62,71#55,72#52,73#55,74#60,75#77,76#59,77#54,78#57,79#65,80#60,81#64,82#62,83#48,84#57,85#51,86#53,87#61,88#62,89#63,90#59,91#53,92#55,93#59,94#50,95#51,96#59,97#58,98#55,99#44,100#47,101#48,102#61,103#47,104#54,105#57,106#50,107#72,108#66,109#53,110#48,111#49,112#51,113#69,114#61,115#59,116#56,117#52,118#61,119#64,120#60,121#56,122#62,123#46,124#50,125#57,126#59,127#58,128#62,129#62,130#54,131#47,132#79,133#52,134#53,135#54,136#52,137#40,138#45,139#54,140#43,141#55,142#56,143#70,144#52,145#58,146#55,147#46,148#54,149#64,150#52,151#56,152#55,153#70,154#45,155#66,156#65,157#58,158#61,159#58,160#57,161#62,162#49,163#76,164#70,165#59,166#68,167#64,168#51,169#68,170#62,171#54,172#51,173#64,174#47,175#59,176#53,177#51,178#44,179#51,180#71,181#60,182#57,183#65,184#68,185#58,186#63,187#51,188#65,189#66,190#47,191#63,192#55,193#56,194#59,195#59,196#53,197#38,198#53,199#47,200#64,201#59,202#56,203#51,204#70,205#50,206#55,207#63,208#42,209#44,210#56,211#50,212#56,213#55,214#56,215#55,216#51,217#57,218#58,219#60,220#47,221#49,222#51,223#42,224#52,225#54,226#56,227#58,228#57,229#53,230#55,231#53,232#47,233#59,234#59,235#58,236#50,237#51,238#54,239#51,240#60,241#53,242#65,243#63,244#60,245#57,246#41,247#64,248#54,249#48,250#49,251#60,252#53,253#53,254#52,255#57,256#62,257#74,258#62,259#57,260#42,261#63,262#52,263#55,264#63,265#47,266#51,267#61,268#49,269#62,270#61,271#63,272#45,273#62,274#53,275#53,276#58,277#58,278#67,279#58,280#59,281#60,282#52,283#43,284#66,285#63,286#50,287#65,288#62,289#55,290#60,291#53,292#50,293#61,294#67,295#69,296#50,297#59,298#50,299#61,300#55,301#61,302#61,303#59,304#51,305#57,306#50,307#60,308#66,309#64,310#66,311#59,312#56,313#51,314#41,315#53,316#54,317#63,318#59,319#54,320#47,321#62,322#57,323#53,324#55,325#54,326#58,327#59,328#42,329#55,330#57,331#53,332#56,333#63,334#55,335#56,336#56,337#52,338#55,339#48,340#50,341#57,342#58,343#47,344#65,345#44,346#70,347#63,348#48,349#59,350#48,351#58,352#57,353#49,354#55,355#55,356#63,357#63,358#59,359#52,360#55,361#60,362#60,363#63,364#62,365#63,366#58,367#68,368#58,369#61,370#64,371#49,372#60,373#62,374#55,375#55,376#52,377#53,378#65,379#58,380#69,381#54,382#53,383#65,384#68,385#61,386#36,387#60,388#55,389#53,390#58,391#53,392#65,393#49,394#52,395#56,396#62,397#50,398#72,399#56,400#52,401#54,402#54,403#63,404#58,405#62,406#56,407#49,408#54,409#61,410#65,411#48,412#60,413#57,414#49,415#61,416#54,417#62,418#53,419#53,420#52,421#51,422#63,423#59,424#48,425#55,426#66,427#56,428#44,429#54,430#61,431#52,432#43,433#44,434#56,435#51,436#60,437#49,438#57,439#54,440#59,441#56,442#54,443#55,444#60,445#58,446#55,447#67,448#53,449#58,450#59,451#58,452#64,453#64,454#62,455#63,456#65,457#60,458#57,459#54,460#47,461#44,462#56,463#58,464#47,465#55,466#44,467#60,468#74,469#68,470#70,471#55,472#60,473#64,474#54,475#56,476#60,477#47,478#63,479#54,480#54,481#55,482#51,483#58,484#47,485#49,486#60,487#52,488#56,489#46,490#43", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0", "mtime": "2020-07-08T23:46:44.705864Z", "creation_time": "2020-07-04T19:41:11.084899Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#", "usage": { "rgw.main": { "size": 69374944756, "size_actual": 70013927424, "size_utilized": 69374944756, "size_kb": 67748970, "size_kb_actual": 68372976, "size_kb_utilized": 67748970, "num_objects": 324880 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } [root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo { "bucket": "df-oldrepo", "num_shards": 967, "tenant": "", "zonegroup": "a1b72aa0-eb06-4a96-8af4-db39d4dc3e09", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732904.1", "marker": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732660.31", "index_type": "Normal", "owner": "01c235fd-fc9b-4c54-a2a5-b38054608b9e", "ver": "0#2,1#2,2#2,3#2,4#2,5#2,6#2,7#2,8#2,9#2,10#2,11#2,12#2,13#2,14#2,15#3,16#2,17#2,18#2,19#2,20#2,21#2,22#3,23#2,24#2,25#2,26#2,27#2,28#2,29#2,30#2,31#2,32#2,33#2,34#2,35#2,36#2,37#2,38#3,39#2,40#2,41#2,42#2,43#2,44#2,45#3,46#2,47#2,48#2,49#2,50#2,51#2,52#2,53#2,54#2,55#2,56#2,57#2,58#2,59#2,60#2,61#2,62#2,63#2,64#2,65#2,66#2,67#2,68#3,69#2,70#2,71#2,72#2,73#2,74#2,75#2,76#2,77#2,78#2,79#2,80#2,81#2,82#2,83#2,84#2,85#2,86#2,87#2,88#2,89#2,90#2,91#2,92#3,93#2,94#2,95#2,96#2,97#2,98#2,99#2,100#2,101#2,102#2,103#2,104#3,105#2,106#2,107#2,108#2,109#2,110#2,111#2,112#2,113#2,114#2,115#2,116#2,117#3,118#2,119#3,120#3,121#2,122#2,123#2,124#2,125#2,126#3,127#2,128#2,129#2,130#2,131#2,132#2,133#2,134#2,135#2,136#2,137#2,138#2,139#2,140#2,141#2,142#2,143#2,144#2,145#2,146#2,147#2,148#2,149#2,150#2,151#2,152#2,153#2,154#2,155#2,156#2,157#2,158#2,159#2,160#2,161#2,162#2,163#2,164#2,165#2,166#2,167#2,168#2,169#2,170#2,171#2,172#2,173#2,174#2,175#2,176#2,177#2,178#2,179#2,180#3,181#2,182#2,183#2,184#2,185#2,186#2,187#2,188#2,189#2,190#2,191#2,192#2,193#2,194#2,195#2,196#2,197#2,198#2,199#2,200#2,201#2,202#2,203#2,204#2,205#2,206#2,207#2,208#2,209#2,210#2,211#2,212#2,213#2,214#2,215#2,216#2,217#2,218#2,219#2,220#2,221#2,222#2,223#2,224#2,225#2,226#2,227#2,228#2,229#2,230#2,231#2,232#2,233#2,234#2,235#2,236#2,237#2,238#2,239#2,240#2,241#2,242#2,243#2,244#2,245#2,246#2,247#2,248#2,249#2,250#2,251#2,252#2,253#2,254#2,255#2,256#2,257#2,258#2,259#2,260#2,261#2,262#2,263#2,264#2,265#2,266#2,267#2,268#2,269#2,270#2,271#2,272#2,273#2,274#2,275#2,276#2,277#2,278#2,279#2,280#2,281#2,282#2,283#2,284#2,285#2,286#2,287#2,288#2,289#2,290#2,291#2,292#2,293#2,294#2,295#2,296#2,297#2,298#2,299#3,300#2,301#2,302#2,303#2,304#2,305#2,306#2,307#2,308#2,309#2,310#2,311#2,312#2,313#2,314#2,315#2,316#2,317#2,318#2,319#2,320#2,321#2,322#2,323#2,324#2,325#2,326#2,327#2,328#2,329#2,330#2,331#2,332#2,333#2,334#2,335#2,336#2,337#2,338#2,339#2,340#2,341#2,342#2,343#2,344#2,345#2,346#2,347#2,348#2,349#2,350#2,351#2,352#2,353#2,354#2,355#2,356#2,357#2,358#2,359#2,360#2,361#2,362#2,363#2,364#2,365#2,366#2,367#2,368#2,369#2,370#2,371#2,372#2,373#2,374#2,375#2,376#2,377#2,378#2,379#2,380#2,381#2,382#2,383#2,384#2,385#2,386#2,387#2,388#2,389#2,390#2,391#2,392#2,393#2,394#2,395#2,396#2,397#2,398#2,399#2,400#2,401#2,402#2,403#2,404#2,405#2,406#2,407#2,408#2,409#2,410#2,411#2,412#2,413#2,414#2,415#2,416#2,417#2,418#2,419#2,420#2,421#2,422#2,423#2,424#2,425#2,426#2,427#2,428#2,429#3,430#2,431#2,432#2,433#2,434#2,435#2,436#2,437#2,438#2,439#2,440#2,441#2,442#2,443#2,444#2,445#2,446#2,447#2,448#2,449#2,450#2,451#2,452#2,453#2,454#2,455#2,456#2,457#2,458#2,459#2,460#2,461#2,462#2,463#2,464#2,465#2,466#2,467#2,468#2,469#2,470#2,471#2,472#2,473#2,474#2,475#2,476#2,477#2,478#2,479#2,480#2,481#2,482#2,483#2,484#2,485#2,486#2,487#2,488#2,489#2,490#2,491#2,492#2,493#2,494#2,495#2,496#2,497#2,498#2,499#2,500#2,501#2,502#2,503#2,504#2,505#2,506#2,507#2,508#3,509#2,510#2,511#2,512#2,513#2,514#2,515#2,516#2,517#2,518#2,519#2,520#2,521#2,522#2,523#2,524#2,525#2,526#2,527#2,528#2,529#2,530#2,531#2,532#2,533#2,534#2,535#2,536#2,537#2,538#2,539#2,540#2,541#2,542#2,543#2,544#2,545#2,546#2,547#2,548#2,549#2,550#2,551#2,552#2,553#2,554#2,555#2,556#2,557#2,558#2,559#2,560#2,561#2,562#2,563#2,564#2,565#2,566#2,567#2,568#2,569#2,570#2,571#2,572#2,573#2,574#2,575#2,576#2,577#2,578#2,579#2,580#2,581#2,582#2,583#2,584#2,585#2,586#2,587#2,588#2,589#2,590#2,591#2,592#2,593#2,594#2,595#2,596#2,597#2,598#2,599#2,600#2,601#2,602#2,603#2,604#2,605#2,606#2,607#2,608#2,609#2,610#2,611#2,612#2,613#2,614#2,615#2,616#2,617#2,618#2,619#2,620#2,621#2,622#2,623#2,624#2,625#2,626#2,627#2,628#2,629#2,630#2,631#2,632#2,633#2,634#2,635#2,636#2,637#2,638#2,639#2,640#2,641#2,642#2,643#2,644#2,645#2,646#2,647#2,648#2,649#2,650#2,651#2,652#2,653#2,654#2,655#2,656#2,657#2,658#2,659#2,660#2,661#2,662#2,663#2,664#2,665#2,666#2,667#2,668#2,669#2,670#2,671#2,672#2,673#2,674#2,675#2,676#2,677#2,678#2,679#2,680#2,681#2,682#2,683#2,684#2,685#2,686#2,687#2,688#2,689#2,690#2,691#2,692#2,693#2,694#2,695#2,696#2,697#2,698#2,699#2,700#2,701#2,702#2,703#2,704#2,705#2,706#2,707#2,708#2,709#2,710#2,711#2,712#2,713#2,714#2,715#2,716#2,717#2,718#2,719#2,720#2,721#2,722#2,723#2,724#2,725#2,726#2,727#2,728#2,729#2,730#2,731#2,732#2,733#2,734#2,735#2,736#2,737#2,738#2,739#2,740#2,741#2,742#2,743#2,744#2,745#2,746#2,747#2,748#2,749#2,750#2,751#2,752#2,753#2,754#2,755#2,756#2,757#2,758#2,759#2,760#2,761#2,762#2,763#2,764#2,765#2,766#2,767#2,768#2,769#2,770#2,771#2,772#2,773#2,774#2,775#2,776#2,777#2,778#2,779#2,780#2,781#2,782#2,783#2,784#2,785#2,786#2,787#2,788#2,789#2,790#2,791#2,792#2,793#2,794#2,795#2,796#2,797#2,798#2,799#2,800#2,801#2,802#2,803#2,804#2,805#2,806#2,807#2,808#2,809#2,810#2,811#2,812#2,813#2,814#2,815#2,816#2,817#2,818#2,819#2,820#2,821#2,822#2,823#2,824#2,825#2,826#2,827#2,828#2,829#2,830#2,831#2,832#2,833#2,834#2,835#2,836#2,837#2,838#2,839#2,840#2,841#2,842#2,843#2,844#2,845#2,846#2,847#2,848#2,849#2,850#2,851#2,852#2,853#2,854#2,855#2,856#2,857#2,858#2,859#2,860#2,861#2,862#2,863#2,864#2,865#2,866#2,867#2,868#2,869#2,870#2,871#2,872#3,873#2,874#2,875#2,876#2,877#2,878#2,879#2,880#2,881#2,882#2,883#2,884#2,885#2,886#2,887#2,888#2,889#2,890#2,891#2,892#2,893#2,894#2,895#2,896#2,897#2,898#2,899#2,900#2,901#2,902#2,903#2,904#2,905#2,906#2,907#2,908#2,909#2,910#2,911#2,912#2,913#2,914#2,915#2,916#2,917#2,918#2,919#2,920#2,921#2,922#2,923#2,924#2,925#2,926#2,927#2,928#2,929#2,930#2,931#2,932#2,933#2,934#2,935#2,936#2,937#2,938#2,939#2,940#2,941#2,942#2,943#2,944#2,945#2,946#2,947#2,948#2,949#2,950#2,951#2,952#2,953#2,954#2,955#2,956#2,957#2,958#2,959#2,960#2,961#2,962#2,963#2,964#2,965#2,966#2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0", "mtime": "2020-07-10T03:28:55.524523Z", "creation_time": "2020-07-08T14:45:49.265385Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#", "usage": { "rgw.main": { "size": 10075673528, "size_actual": 10174205952, "size_utilized": 10075673528, "size_kb": 9839525, "size_kb_actual": 9935748, "size_kb_utilized": 9839525, "num_objects": 47476 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } [root@os1 ~]# radosgw-admin bucket stats --bucket=df-test { "bucket": "df-test", "num_shards": 11, "tenant": "", "zonegroup": "a1b72aa0-eb06-4a96-8af4-db39d4dc3e09", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732580.22", "marker": "d8c6ebd1-2bab-414d-9d6b-73bf9bc8fc5a.9732580.22", "index_type": "Normal", "owner": "01c235fd-fc9b-4c54-a2a5-b38054608b9e", "ver": "0#1,1#3,2#1,3#1,4#1,5#1,6#1,7#1,8#3,9#1,10#2", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "2020-07-08T14:33:11.589606Z", "creation_time": "2020-07-08T14:33:11.575423Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 511, "size_actual": 4096, "size_utilized": 511, "size_kb": 1, "size_kb_actual": 4, "size_kb_utilized": 1, "num_objects": 1 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } [root@os1 ~]# radosgw-admin user stats --uid=user-in-question { "stats": { "size": 11154608110526, "size_actual": 11263376089088, "size_utilized": 0, "size_kb": 10893171983, "size_kb_actual": 10999390712, "size_kb_utilized": 0, "num_objects": 52543794 }, "last_stats_sync": "2020-07-14T12:08:58.888165Z", "last_stats_update": "2020-07-14T12:08:58.887088Z" } Cheers, Dave

1 0

User stats - Object count wrong in Octopus?
by David Monschein 14 Jul '20

14 Jul '20

Hi All, I'm investigating what appears to be a bug in RGW stats. This is a brand new cluster running 15.2.3 One of our customers reached out, saying they were hitting their quota (S3 error: 403 (QuotaExceeded)). The user-wide max_objects quota we set is 50 million objects, so this would be impossible since the entire cluster isn't even close to 50 million objects yet: [root@os1 ~]# ceph status | grep objects objects: 7.58M objects, 6.8 TiB The customer in question has three buckets, and if I query the bucket stats, the total number of objects for all 3 buckets comes to about ~372k: [root@os1 ~]# radosgw-admin bucket stats --bucket=df-fs1 | grep num_objects "num_objects": 324880 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-oldrepo | grep num_objects "num_objects": 47476 [root@os1 ~]# radosgw-admin bucket stats --bucket=df-test | grep num_objects "num_objects": 1 But things get interesting when I query the user stats:

1 0

cephadm adoption failed
by Tobias Gall 14 Jul '20

14 Jul '20

Hello, I'm trying to adopt an existing cluster. The cluster consists of 5 converged (mon, mgr, osd, mds on same host) servers running Octopus 15.2.4. I've followed the guide: https://docs.ceph.com/docs/octopus/cephadm/adoption/ Adopting the first mon I've got the following problem: root@mulberry:/home/toga# cephadm adopt --style legacy --name mon.mulberry INFO:cephadm:Pulling latest docker.io/ceph/ceph:v15 container... INFO:cephadm:Stopping old systemd unit ceph-mon@mulberry... INFO:cephadm:Disabling old systemd unit ceph-mon@mulberry... INFO:cephadm:Moving data... INFO:cephadm:Chowning content... Traceback (most recent call last): File "/usr/sbin/cephadm", line 4761, in <module> r = args.func() File "/usr/sbin/cephadm", line 1162, in _default_image return func() File "/usr/sbin/cephadm", line 3241, in command_adopt command_adopt_ceph(daemon_type, daemon_id, fsid); File "/usr/sbin/cephadm", line 3387, in command_adopt_ceph call_throws(['chown', '-c', '-R', '%d.%d' % (uid, gid), data_dir_dst]) File "/usr/sbin/cephadm", line 844, in call_throws out, err, ret = call(command, **kwargs) File "/usr/sbin/cephadm", line 784, in call message = message_b.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 1023: unexpected end of data In `cephadm ls` the old mon is gone and the new is present: { "style": "cephadm:v1", "name": "mon.mulberry", "fsid": "74307e84-e1fe-4706-8312-fe47703928a1", "systemd_unit": "ceph-74307e84-e1fe-4706-8312-fe47703928a1(a)mon.mulberry", "enabled": false, "state": "stopped", "container_id": null, "container_image_name": null, "container_image_id": null, "version": null, "started": null, "created": null, "deployed": null, "configured": null } But there is no container running. How can I resolve this? Regards, Tobias

2 4