- ceph-users - lists.ceph.io

ceph-df free discrepancy
by Reed Dier 11 Apr '20

11 Apr '20

Hopefully someone can sanity check me here, but I'm getting the feeling that the MAX AVAIL in ceph df isn't reporting the correct value in 14.2.8 (mon/mgr/mds are .8, most OSDs are .7) > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 530 TiB 163 TiB 366 TiB 367 TiB 69.19 > ssd 107 TiB 37 TiB 70 TiB 70 TiB 65.33 > TOTAL 637 TiB 201 TiB 436 TiB 437 TiB 68.54 > > POOLS: > POOL ID STORED OBJECTS USED %USED MAX AVAIL > fs-metadata 16 44 GiB 4.16M 44 GiB 0.25 5.6 TiB > cephfs-hdd-3x 17 46 TiB 109.54M 144 TiB 61.81 30 TiB > objects-hybrid 20 46 TiB 537.08M 187 TiB 91.71 5.6 TiB > objects-hdd 24 224 GiB 50.81k 676 GiB 0.74 30 TiB > rbd-hybrid 29 3.8 TiB 1.19M 11 TiB 40.38 5.6 TiB > device_health_metrics 33 270 MiB 327 270 MiB 0 30 TiB > rbd-ssd 34 4.2 TiB 1.19M 12 TiB 41.55 5.6 TiB > cephfs-hdd-ec73 37 42 TiB 30.35M 72 TiB 44.78 62 TiB I have a few pools that don't look like they are calculating the available storage for that pool correctly. Specifically, any of my hybrid pools (20,29) or all-SSD pools (16,34). For my hybrid pools, I have a crush rule of take 1 of host in the ssd root, take -1 chassis in the hdd root. For my ssd pools, I have a crush rule of take 0 of host in the ssd root. Now I have 60 ssd osds 1.92T each, and sadly distribution is imperfect (leaving those issues out of this), and I have plenty of underfull and overfull osds, which I am trying to manually reweighs to get my most full's down to free up space: > $ ceph osd df class ssd | sort -k17 > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS > MIN/MAX VAR: 0.80/1.21 STDDEV: 5.56 > TOTAL 107 TiB 70 TiB 68 TiB 163 GiB 431 GiB 37 TiB 65.33 > 28 ssd 1.77879 1.00000 1.8 TiB 951 GiB 916 GiB 7.1 GiB 5.8 GiB 871 GiB 52.20 0.80 68 up > 33 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1010 GiB 6.0 MiB 5.9 GiB 777 GiB 57.33 0.88 74 up > 47 ssd 1.77879 1.00000 1.8 TiB 1.0 TiB 1011 GiB 6.7 GiB 6.4 GiB 776 GiB 57.38 0.88 75 up > [SNIP] > 57 ssd 1.77879 0.98000 1.8 TiB 1.4 TiB 1.3 TiB 6.2 GiB 8.6 GiB 417 GiB 77.08 1.18 102 up > 107 ssd 1.80429 1.00000 1.8 TiB 1.4 TiB 1.3 TiB 7.0 GiB 8.7 GiB 422 GiB 77.15 1.18 102 up > 50 ssd 1.77879 1.00000 1.8 TiB 1.4 TiB 1.4 TiB 5.5 MiB 8.6 GiB 381 GiB 79.10 1.21 105 up > 60 ssd 1.77879 0.92000 1.8 TiB 1.4 TiB 1.4 TiB 6.2 MiB 9.0 GiB 379 GiB 79.17 1.21 105 up That said, as a straw man argument, ~380GiB free, times 60 OSDs, should be ~22.8TiB free, if all OSD's grew evenly, which they won't, which is still far short of 37TiB raw free, as expected. However, what doesn't track is the 5.6TiB available at the pools level, even for a 3x replicated pool (5.6*3=16.8TiB, which is about 34% less than my napkin math, which would be 22.8/3=7.6TiB. But what tracks even less is the hybrid pools, which use 1/3 of what the 3x-replicated data consumes. Meaning if my napkin math is right, should show ~22.8TiB free. Am I grossly mis-understanding how this is calculated? Maybe this is fixed in Octopus? Just trying to get a grasp on what I'm seeing not matching expectations. Thanks, Reed

2 2

remove S3 bucket with rados CLI
by Robert Sander 10 Apr '20

10 Apr '20

Hi, is it possible to remove an S3 bucket and its S3 objects with the rados CLI tool (removing the low level Ceph objects)? The situation is a nearfull cluster with OSDs filled more than 80% and the default.rgw.buckets.data being reported as 100% full. Read operations are still possible but no write. s3cmd rb … says operation halted and radosgw-admin bucket rm also waits for a healthy cluster. The other option would be to tune the nearfull_ratio and full_ratio temporarily to allow the cluster to use more space, correct? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 4

Understanding monitor requirements
by Brian Topping 10 Apr '20

10 Apr '20

Hi experts, question about monitors and latency. I am setting up a new cluster and I’d like to have more than one monitor. Unfortunately, the primary site only has two chassis, so to get the third mon, I’ve been trying to bring it up remotely. So far, it’s not working and I wonder if someone knows how I can fit the proverbial square peg in this round hole. This gist[1] has the logs that I have from the third (remote) monitor and a diagram of the network. The logs keep repeating the same errors about slow ops from the first log entry. For comparison, I’ve also included a performance test of etcd that is running on the same three nodes. The secondary site node is slower by an order of magnitude. As for the use case, I mostly expect the primary hardware to be stable and the third mon provides uptime when I need to upgrade either primary node. There will never be any OSD traffic over the site-to-site link. The storage profiles are primarily RBD with static sizes and limited changes. My understanding is that the monitors are only keeping track of PG locations for pools. If that’s the case and the storage is primarily fixed size RBDs, the amount of inter-monitor communication should also be limited since changes are few. Am I mistaken here? Thanks for any input that can be provided! Brian [1] https://gist.github.com/briantopping/3638d665bc60c19e6f3f2005eb555a9d

1 1

Profiling of CRUSH Ceph
by Bobby 09 Apr '20

09 Apr '20

Hi all, I am a novice Ceph user and I am going through Ceph CRUSH algorithm. I would like to do some profiling of CRUSH computations on a CPU. Are there any CRUSH benchmarks? Any profiling tool for CRUSH? I would like to examine what are the compute intensive parts of CRUSH on the CPU. May be there is some research paper on it already? The goal is hardware and software partioning. I am really looking forward for some helping tips. Bobby !

1 0

v15.2.1 Octopus released
by Abhishek 09 Apr '20

09 Apr '20

This is the first bugfix release of Ceph Octopus, we recommend all Octopus users upgrade. This release fixes an upgrade issue and also has 2 security fixes Notable Changes ~~~~~~~~~~~~~~~ * issue#44759: Fixed luminous->nautilus->octopus upgrade asserts * CVE-2020-1759: Fixed nonce reuse in msgr V2 secure mode * CVE-2020-1760: Fixed XSS due to RGW GetObject header-splitting For the full changelog please refer to the official release blog entry at https://ceph.io/releases/v15-2-1-octopus-released/, for upgrades from previous versions, please follow the relevant section of the upgrade docs at https://docs.ceph.com/docs/master/releases/octopus/#upgrading-from-mimic-or… Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.1.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 9fd2f65f91d9246fae2c841a6222d34d121680ee

1 0

Balancer not balancing (14.2.7, crush-compat)
by Vladimir Brik 09 Apr '20

09 Apr '20

Hello I am running ceph 14.2.7 with balancer in crush-compat mode (needed because of old clients), but it's doesn't seem to be doing anything. It used to work in the past. I am not sure what changed. I created a big pool, ~285TB stored, and it doesn't look like it ever got balanced: pool 43 'fs-data-k5m2-hdd' erasure size 7 min_size 6 crush_rule 7 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode warn last_change 48647 lfor 0/42080/42102 flags hashpspool,ec_overwrites,nearfull stripe_width 20480 application cephfs OSD utilization varies between ~50% and about ~80%, with about 60% raw used. I am using a mixture of 9TB and 14TB drives. Number of PGs/drive varies 103 and 207. # ceph osd df | grep hdd | sort -k 17 | (head -n 2; tail -n 2) 160 hdd 12.53519 1.00000 13 TiB 6.0 TiB 5.9 TiB 74 KiB 12 GiB 6.6 TiB 47.74 0.79 120 up 146 hdd 12.53519 1.00000 13 TiB 6.0 TiB 6.0 TiB 51 MiB 13 GiB 6.5 TiB 48.17 0.80 119 up 79 hdd 8.99799 1.00000 9.0 TiB 7.3 TiB 7.2 TiB 42 KiB 16 GiB 1.7 TiB 80.91 1.34 186 up 62 hdd 8.99799 1.00000 9.0 TiB 7.3 TiB 7.2 TiB 112 KiB 16 GiB 1.7 TiB 81.44 1.35 189 up # ceph balancer status { "last_optimize_duration": "0:00:00.339635", "plans": [], "mode": "crush-compat", "active": true, "optimize_result": "Some osds belong to multiple subtrees: {0: ['default', 'default~hdd'], ... "last_optimize_started": "Thu Apr 9 11:17:40 2020" } Does anybody know how to debug this? Thanks, Vlad

1 1

Bucket index entries containing unicode NULL - how to remove them?
by Maks Kowalik 09 Apr '20

09 Apr '20

Hello, since some time I've been investigating problems causing buckets' index corruption. In my case it's been because of numerous bugs related to index resharding and bucket lifecycle policies. One of those bugs present in versions prior to 14.2.8 made the index omapkeys' names contain unicode NULL which manifests like this: root@mach0122:~/mkw # radosgw-admin bi list --bucket=sysa-user-logs|grep idx |tail -5 "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031521.gz.u6_qPM0X2mXQRXD0hEfRK2dCc4dD4El\u0000.4", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031600.gz.bvZdIu5KnsAQyXoa_ZWpix-BYD-yvcz\u0000.15", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031600.gz.wX9jHyKnJ21w5DYL_rdfTsb7zH3tUEa\u0000.10", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031601.gz.K6GhZfNzcqDCcjkrn5GskRWS8ufXSbO\u0000.8", "idx": "_multipart_user-ec24b85efa1c/user-ec24b85efa1c-AA-U79.log-2020031601.gz.vEr4VCu0QU07te_RHNJ9Wi1cb8tsiYq\u0000.7", I am unable to remove them by rmomapkey. Also I tried to use the method https://github.com/ceph/ceph/blob/8c63b26fe88bb02d894705cb1beec289668fb43d/… but it fails with: File "./idx_removal.py", line 96, in remove_key iocontext.remove_omap_keys(write_op, key) File "rados.pyx", line 516, in rados.requires.wrapper.validate_func (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:4992) File "rados.pyx", line 3607, in rados.Ioctx.remove_omap_keys (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:45059) File "rados.pyx", line 543, in rados.cstr_list (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:5665) File "rados.pyx", line 539, in rados.cstr (/build/ceph-14.2.8/obj-x86_64-linux-gnu/src/pybind/rados/pyrex/rados.c:5463) TypeError: keys must be a string Does anyone know what else can be done to remove such omapkey? For the obvious reasons I can't use the clear_omap method... Kind regards, Maks Kowalik

1 0

Using M2 SSDs as osds
by Stolte, Felix 09 Apr '20

09 Apr '20

Hey guys, I am evaluating using M2 SSDs as osds for an all flash pool. Is anyone using that in production and can elaborate on his experience? I am a little bit concerned about the lifetime of the M2 disks. Best regards Felix IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

2 1

Nautilus 14.2.7 radosgw lifecycle not removing expired objects
by oneill.gs＠gmail.com 09 Apr '20

09 Apr '20

Hello, I'm running a cluster with Ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable). I've encountered an issue with my cluster where objects are marked as expired but are not removed during lifecycle processing. These buckets have a mix of objects with and without special characters in their names. I set debug_rgw to 20/5 overnight so I could get more detailed logs when the lifecycle processing happened. The lifecycle function seems to correctly identify expired objects that should be removed, however they are not deleted. This is happening across multiple buckets and users. This cluster does have dynamic bucket index resharding enabled, however this issue affects buckets regardless of their reshard status. I also have cache tiering set to readproxy for the buckets data pool. root@rgw01 # ceph daemon /var/run/ceph/ceph-rgw-rgw01.asok config show | grep -E "lifecycle|lc" "rgw_lifecycle_work_time": "00:00-06:00", "rgw_enable_lc_threads": "true", "rgw_lc_debug_interval": "-1", "rgw_lc_lock_max_time": "60", "rgw_lc_max_objs": "32", "rgw_lc_max_rules": "1000", "rgw_lc_thread_delay": "0", root@mon01 # radosgw-admin lc list [ { "bucket": ":testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50", "status": "COMPLETE" } ] root@mon01 # radosgw-admin lc get --bucket testgso { "prefix_map": { "": { "status": true, "dm_expiration": false, "expiration": 1, "noncur_expiration": 0, "mp_expiration": 1 } }, "rule_map": [ { "id": "111222333", "rule": { "id": "111222333", "prefix": "", "status": "Enabled", "expiration": { "days": "1", "date": "" }, "noncur_expiration": { "days": "", "date": "" }, "mp_expiration": { "days": "1", "date": "" }, "filter": { "prefix": "", "obj_tags": {} }, "dm_expiration": false } } ] } goneill@mbp $ s3cmd -c Documents/s3cfg/testgso.cfg ls s3://testgso DIR s3://testgso/lifecycle/ 2020-03-31 17:26 12 s3://testgso/file!test.txt 2020-03-31 17:29 12 s3://testgso/file$test.txt 2020-03-31 17:29 12 s3://testgso/file&test.txt 2020-03-31 17:26 12 s3://testgso/file'test.txt 2020-03-31 17:26 12 s3://testgso/file(test.txt 2020-03-31 17:26 12 s3://testgso/file)test.txt 2020-03-31 17:29 12 s3://testgso/file*test.txt 2020-03-31 17:30 12 s3://testgso/file+test.txt 2020-03-31 17:30 12 s3://testgso/file,test.txt 2020-03-31 17:26 12 s3://testgso/file-test.txt 2020-03-31 17:30 12 s3://testgso/file:test.txt 2020-03-31 17:30 12 s3://testgso/file;test.txt 2020-03-31 17:30 12 s3://testgso/file=test.txt 2020-03-31 17:30 12 s3://testgso/file?test.txt 2020-03-31 17:06 12 s3://testgso/file@test.txt 2019-08-05 21:02 12 s3://testgso/test.txt goneill@mbp $ s3cmd -c Documents/s3cfg/testgso.cfg getlifecycle s3://testgso <?xml version="1.0" ?> <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Rule> <ID>111222333</ID> <Prefix/> <Status>Enabled</Status> <Expiration> <Days>1</Days> </Expiration> <AbortIncompleteMultipartUpload> <DaysAfterInitiation>1</DaysAfterInitiation> </AbortIncompleteMultipartUpload> </Rule> </LifecycleConfiguration> ### Logs from rgw01 2020-04-08 00:00:00.645 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc9bf0 obj=default.rgw.meta:root:testgso state=0x561a8bc0ac40 s->prefetch_data=0 2020-04-08 00:00:00.645 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+testgso : expiry miss 2020-04-08 00:00:00.645 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+testgso info.flags=0x16 2020-04-08 00:00:00.645 7fc55bbcc700 10 adding default.rgw.meta+root+testgso to cache LRU end 2020-04-08 00:00:00.645 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.645 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.645 7fc55bbcc700 20 Read xattr: user.rgw.idtag 2020-04-08 00:00:00.645 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+testgso : type miss (requested=0x13, cached=0x16) 2020-04-08 00:00:00.645 7fc55bbcc700 20 rados->read ofs=0 len=0 2020-04-08 00:00:00.646 7fc55bbcc700 20 rados_obj.operate() r=0 bl.length=181 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+testgso info.flags=0x13 2020-04-08 00:00:00.646 7fc55bbcc700 10 moving default.rgw.meta+root+testgso to cache LRU end 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.646 7fc55bbcc700 20 rgw_get_bucket_info: bucket instance: testgso[0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50] 2020-04-08 00:00:00.646 7fc55bbcc700 20 reading from default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.646 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc9bf0 obj=default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 state=0x561a9d27d540 s->prefetch_data=0 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : expiry miss 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 info.flags=0x16 2020-04-08 00:00:00.646 7fc55bbcc700 10 adding default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 to cache LRU end 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.acl bl.length()=153 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.lc bl.length()=130 2020-04-08 00:00:00.646 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.646 7fc55bbcc700 20 Read xattr: user.rgw.acl 2020-04-08 00:00:00.646 7fc55bbcc700 20 Read xattr: user.rgw.idtag 2020-04-08 00:00:00.646 7fc55bbcc700 20 Read xattr: user.rgw.lc 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : type miss (requested=0x13, cached=0x16) 2020-04-08 00:00:00.646 7fc55bbcc700 20 rados->read ofs=0 len=0 2020-04-08 00:00:00.646 7fc55bbcc700 20 rados_obj.operate() r=0 bl.length=281 2020-04-08 00:00:00.646 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 info.flags=0x13 2020-04-08 00:00:00.646 7fc55bbcc700 10 moving default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 to cache LRU end 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.acl bl.length()=153 2020-04-08 00:00:00.646 7fc55bbcc700 10 updating xattr: name=user.rgw.lc bl.length()=130 2020-04-08 00:00:00.646 7fc55bbcc700 10 chain_cache_entry: cache_locator=default.rgw.meta+root+testgso 2020-04-08 00:00:00.646 7fc55bbcc700 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.647 7fc55bbcc700 10 lifecycle: bucket_lc_process() prefix_map size=1 2020-04-08 00:00:00.647 7fc55bbcc700 20 lifecycle: bucket_lc_process(): prefix= 2020-04-08 00:00:00.647 7fc55bbcc700 10 cls_bucket_list_unordered testgso[0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50] start [] num_entries 1100 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file!test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file$test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file&test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file'test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file(test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file)test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file*test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file+test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file,test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file-test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file:test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file;test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file=test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file?test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got file(a)test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got lifecycle/file.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 10 RGWRados::cls_bucket_list_unordered: got test.txt[] 2020-04-08 00:00:00.647 7fc55bbcc700 20 lifecycle: bucket_lc_process(): key=file!test.txt 2020-04-08 00:00:00.648 7fc55bbcc700 20 obj_has_expired(): mtime=2020-03-31 13:26:07.0.481384s days=1 base_time=2020-04-08 00:00:00.000000 timediff=642833 cmp=86400 2020-04-08 00:00:00.648 7fc55bbcc700 20 check(): key=file!test.txt: is_expired=1 2020-04-08 00:00:00.648 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file!test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.657 7fc44f1b3700 10 manifest: total_size = 49667 2020-04-08 00:00:00.657 7fc44f1b3700 20 get_obj_state: setting s->obj_tag to 0a6e5079-ad65-4ea7-a868-b2133099e0d0.22351662.19462946 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_obj_state: setting s->obj_tag to 0a6e5079-ad65-4ea7-a868-b2133099e0d0.25454121.7727201 2020-04-08 00:00:00.672 7fc55bbcc700 10 If-UnModified-Since: 2020-03-31 13:26:07.0.481384s Last-Modified: 2020-03-31 13:26:07.000000 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file!test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.672 7fc55bbcc700 20 prepare_atomic_modification: state is not atomic. state=0x561aa0c64aa0 2020-04-08 00:00:00.672 7fc55bbcc700 20 reading from default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc8330 obj=default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 state=0x561a9d27d0c0 s->prefetch_data=0 2020-04-08 00:00:00.672 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : type miss (requested=0x16, cached=0x13) 2020-04-08 00:00:00.672 7fc55bbcc700 10 cache put: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 info.flags=0x16 2020-04-08 00:00:00.672 7fc55bbcc700 10 moving default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 to cache LRU end 2020-04-08 00:00:00.672 7fc55bbcc700 10 updating xattr: name=ceph.objclass.version bl.length()=42 2020-04-08 00:00:00.672 7fc55bbcc700 10 updating xattr: name=user.rgw.acl bl.length()=153 2020-04-08 00:00:00.672 7fc55bbcc700 10 updating xattr: name=user.rgw.lc bl.length()=130 2020-04-08 00:00:00.672 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.672 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : hit (requested=0x11, cached=0x17) 2020-04-08 00:00:00.672 7fc55bbcc700 20 bucket index object: .dir.0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.683 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.683 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.683 7fc55bbcc700 20 lifecycle: ERROR: orule.process() returned ret=-125 2020-04-08 00:00:00.683 7fc55bbcc700 20 lifecycle: bucket_lc_process(): key=file$test.txt 2020-04-08 00:00:00.683 7fc55bbcc700 20 obj_has_expired(): mtime=2020-03-31 13:29:57.0.916797s days=1 base_time=2020-04-08 00:00:00.000000 timediff=642603 cmp=86400 2020-04-08 00:00:00.683 7fc55bbcc700 20 check(): key=file$test.txt: is_expired=1 2020-04-08 00:00:00.683 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file$test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.684 7fc55bbcc700 10 manifest: total_size = 12 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_obj_state: setting s->obj_tag to 0a6e5079-ad65-4ea7-a868-b2133099e0d0.25454121.7731015 2020-04-08 00:00:00.684 7fc55bbcc700 10 If-UnModified-Since: 2020-03-31 13:29:57.0.916797s Last-Modified: 2020-03-31 13:29:57.000000 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_obj_state: rctx=0x7fc55bbc8ff8 obj=testgso:file$test.txt state=0x561aa0c64aa0 s->prefetch_data=0 2020-04-08 00:00:00.684 7fc55bbcc700 20 prepare_atomic_modification: state is not atomic. state=0x561aa0c64aa0 2020-04-08 00:00:00.684 7fc55bbcc700 20 reading from default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_system_obj_state: rctx=0x7fc55bbc8330 obj=default.rgw.meta:root:.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 state=0x561a9d27d0c0 s->prefetch_data=0 2020-04-08 00:00:00.684 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : hit (requested=0x16, cached=0x17) 2020-04-08 00:00:00.684 7fc55bbcc700 20 get_system_obj_state: s->obj_tag was set empty 2020-04-08 00:00:00.684 7fc55bbcc700 10 cache get: name=default.rgw.meta+root+.bucket.meta.testgso:0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 : hit (requested=0x11, cached=0x17) 2020-04-08 00:00:00.684 7fc55bbcc700 20 bucket index object: .dir.0a6e5079-ad65-4ea7-a868-b2133099e0d0.5495934.50 2020-04-08 00:00:00.711 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.711 7fc55bbcc700 0 ERROR: remove_expired_obj 2020-04-08 00:00:00.711 7fc55bbcc700 20 lifecycle: ERROR: orule.process() returned ret=-125

1 0

[Octopus] OSD overloading
by Jack 08 Apr '20

08 Apr '20

Hello, I've a issue, since my Nautilus -> Octopus upgrade My cluster has many rbd images (~3k or something) Each of them has ~30 snapshots Each day, I create and remove a least a snapshot per image Since Octopus, when I remove the "nosnaptrim" flags, each OSDs uses 100% of its CPU time The whole cluster collapses: OSDs no longer see each others, most of them are seens as down .. I do not see any progress being made : it does not appear the problem will solve by itself What can I do ? Best regards,

8 12