- ceph-users - lists.ceph.io

Re: Multisite RGW - Large omap objects related with bilogs
by Scheurer François 23 Mar '21

23 Mar '21

Dear All We have the same question here, if anyone can help ... Thank you! Cheers Francois ________________________________ From: ceph-users on behalf of P. O. <posdub(a)gmail.com> Sent: Friday, August 9, 2019 11:05 AM To: ceph-users(a)lists.ceph.com Subject: [ceph-users] Multisite RGW - Large omap objects related with bilogs Hi all, I have two ceph clusters in RGW multisite environment, with ~1500 bucketes ( 500M objects, 70TB ). Some of the buckets are very dynamic (objects are constantly changing). I have problems with large omap objects in bucket indexes, related with "dynamic buckets". For example: [root@rgw ~]# radosgw-admin bucket stats --bucket bucket_s3d33 |grep num_objects "num_objects": 564 In /var/log/ceph/ceph.log: cluster [WRN] Large omap object found. Object: 10:297646ca:::.dir.86a05ec8-9982-429b-9f94-28363610a95c.12546d0.17892:head Key count: 5307523 Size (bytes): 748792509 I found, this is because of bucket index logs: [root@rgw-1 ~]# rados -p default.rgw.buckets.index listomapkeys .dir.86a05ec8-9982-429b-9f94-28363610a95c.12546d0.17892 | wc -l 5307523 There are a lot of keys: �0_00013758656.71188336.4 �0_00013758657.71188337.5 �0_00013758658.71188338.4 �0_00013758659.71188339.5 �0_00013758660.71188342.4 �0_00013758661.71188343.5 �0_00013758662.71188344.4 [root@rgw-1 ~]# radosgw-admin bilog list --bucket bucket_s3d33 --max-entries 6000000 |grep op_id | wc -l 5307523 I have configured parameters in my ceph.conf: rgw sync log trim concurrent buckets = 32 rgw sync log trim max buckets = 64 rgw sync log trim interval = 1200 rgw sync log trim min cold buckets = 4 But from two weeks, the omap key count is still growing. How can I safely clean these bilogs (with no bucket damage and no replication damage)? I found two radosgw-admin parameters related with bilogs trimming: 1) radosgw-admin bilog trim --bucket=bucket_s3d33 --start-marker XXXX --end-marker YYYY I dont know what values should be in: --start-marker XXXX --end-marker YYYY. Is it safe to use "bilog trim" on bucket with replication in progress? If yes, should i run this on both sites? 2) radosgw-admin bilog autotrim Is this command safe? Can I use autotrim on selected bucket? Maybe there is some other way, to delete bilogs? Best regards, P.O. << ATT00001.txt (0.4KB) (0.4KB) >>

1 0

DocuBetter Meeting -- APAC 25 Mar 2021 0100 UTC
by John Zachary Dover 22 Mar '21

22 Mar '21

There will be a DocuBetter meeting on Thursday, 25 Mar 2021 at 0100 UTC. We will discuss the Google Season of Docs proposal (the Comprehensive Contribution Guide), the rewriting of the cephadm documentation and the new sectin of the Teuthology Guide. DocuBetter Meeting -- APAC 25 Mar 2021 0100 UTC https://bluejeans.com/908675367 https://pad.ceph.com/p/Ceph_Documentation

1 0

March 2021 Tech Talk and Code Walk-through
by Mike Perez 22 Mar '21

22 Mar '21

Hi everyone! I'm excited to announce two talks we have on the schedule for March 2021: Persistent Bucket Notifications By Yuval Lifshitz https://ceph.io/ceph-tech-talks/ The stream starts on March 25th at 17:00 UTC / 18:00 CET / 1:00 PM EST / 10:00 AM PST Persistent bucket notifications are going to be introduced in Ceph "Pacific." The idea behind them is to allow for reliable and asynchronous delivery of notifications from the RADOS gateway (RGW) to the endpoint configured at the topic. Regular notifications could also be considered reliable since the delivery to the endpoint is performed synchronously during the request. However, this reliability is only from the RGW perspective, meaning that the client will not get an ACK until an ACK is received from the endpoint but does not retry if the endpoint is down or disconnected. Also, note that, with regular notifications, if the endpoint sends back a NACK, the operation is still considered successful (since there is no way to rollback the RADOS operations that happened before the notification was tried). When the endpoint is down but failed to push, the notification is only timeout base; using regular notifications will slow down the operation of the RGW and may bring it to a complete halt. With persistent notifications, we allow the RGW to retry sending notifications even if the endpoint is down or a network disconnect with it during the operation (notifications are retried if not successfully delivered to the endpoint). Also, the operation is a-synchronous, so during the operation, the notifications are just pushed into a queue (see below), and the actual sending to the endpoint is happening a-synchronously. The queuing operation is done in 2 phases (reserve and commit or abort) to guarantee the queuing operation's atomicity with the other operations. In case you missed the last Tech Talk, see Sage Weil's presentation on What's new in Pacific: https://www.youtube.com/watch?v=PVtn53MbxTc -------------- Samuel Just will be giving a code walkthrough on RADOS Snapshots. https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs The stream starts on March 23rd at 17:00 UTC / 18:00 CET / 1:00 PM EST / 10:00 AM PST In case you missed it, watch Part 2 of LibRBD I/O Flow by Jason Dillaman: https://www.youtube.com/watch?v=nVjYVmqNClM All live streams will be recorded. -- Mike Perez (thingee)

1 0

how to disable write-back mode in ceph octopus
by 无名万剑归宗 22 Mar '21

22 Mar '21

I tried cache tier in write-back mode in my cluster, but because my ssd drive is home used, can not satisfy the needs of IOPS. Now I want disable write-back mode , I founded office documents,but the doc was outdated https://docs.ceph.com/en/latest/rados/operations/cache-tiering/?highlight=c… ）. root@e9000-22:~# ceph osd tier cache-mode cache proxy > Invalid command: proxy not in writeback|readproxy|readonly|none > osd tier cache-mode <pool> writeback|readproxy|readonly|none > [--yes-i-really-mean-it] : specify the caching mode for cache tier <pool> > Error EINVAL: invalid command Anyone can tell my how to disable this mode.

1 0

Question about migrating from iSCSI to RBD
by Justin Goetz 22 Mar '21

22 Mar '21

Hello! I was hoping to inquire if anyone here has attempted similar operations, and if they ran into any issues. To give a brief overview of my situation, I have a standard octopus cluster running 15.2.2, with ceph-iscsi installed via ansible. The original scope of a project we were working on changed, and we no longer need the iSCSI overhead added to the project (the machine using CEPH is Linux, so we would like to use native RBD block devices instead). Ideally we would create some new pools and migrate the data from the iSCSI pools over to the new pools, however, due to the massive amount of data (close to 200 TB), we lack the physical resources necessary to copy the files. Digging a bit on the backend of the pools utilized by ceph-iscsi, it appears that the iSCSI utility uses standard RBD images on the actual backend: ~]# rbd info iscsi/pool-name rbd image 'pool-name': size 200 TiB in 52428800 objects order 22 (4 MiB objects) snapshot_count: 0 id: 137b45a37ad84a block_name_prefix: rbd_data.137b45a37ad84a format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: object map invalid, fast diff invalid create_timestamp: Thu Nov 12 16:14:31 2020 access_timestamp: Tue Mar 16 16:13:41 2021 modify_timestamp: Tue Mar 16 16:15:36 2021 And I can also see that, like a standard rbd image, our 1st iSCSI gateway currently holds the lock on the image: ]# rbd lock ls --pool iscsi pool-name There is 1 exclusive lock on this image. Locker ID Address client.3618592 auto 259361792 10.101.12.61:0/1613659642 Theoretically speaking, would I be able to simply stop & disable the tcmu-runner processes on all iSCSI gateways in our cluster, which would release the lock on the RBD image, then create another user with rwx permissions to the iscsi pool? Would this work, or am I missing something that would come back to bite me later on? Looking for any advice on this topic. Thanks in advance for reading! -- Justin Goetz Systems Engineer, TeraSwitch Inc. jgoetz(a)teraswitch.com 412-945-7045 (NOC) | 412-459-7945 (Direct)

2 2

How to sizing nfs-ganesha.
by Quang Lê 22 Mar '21

22 Mar '21

Hi guys, I'm using manila-openstack to provide a filesystem service using backend CEPHFS. My design use nfs-ganesha as the gateway for the VM in openstack mount to CephFS. I am having problems with sizing the ganesha-servers. Can anyone suggest me *what are the hardware requirements of the ganesha server are* or *what parameters are needed to consider when sizing a ganesha server* ? My simple topology in link: *https://i.imgur.com/xrYqxAh.png <https://i.imgur.com/xrYqxAh.png>* Thank you guys.

2 1

Incomplete pg , any chance to to make it survive or data loss :( ?
by Szabo, Istvan (Agoda) 22 Mar '21

22 Mar '21

Hi, What can I do with this pg to make it work? We lost and don't have the osds 61,122 but we have the 32,33,70. I've exported the pg chunk from them, but they are very small and when I imported back to another osd that osd never started again so I had to remove that chunk (44.1aas2, 44.1aas3) to be able to start the osd. [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg incomplete pg 44.1aa is incomplete, acting [59,128,127,43] (reducing pool cephfs1-data01-pool min_size from 3 may help; search ceph.com/docs for 'incomplete') [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100 [WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time pg 44.1aa not scrubbed since 2021-01-14T05:50:23.852626+0100 [WRN] SLOW_OPS: 96 slow ops, oldest one blocked for 228287 sec, osd.59 has slow ops This is the pg query and pg map important parts: "probing_osds": [ "29(3)", "34(3)", "43(3)", "56(1)", "59(0)", "72(2)", "73(2)", "74(2)", "127(2)", "128(1)", "131(2)" ], "down_osds_we_would_probe": [ 32, 33, 61, 70, 122 ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { "detail": "peering_blocked_by_history_les_bound" osdmap e5666778 pg 44.1aa (44.1aa) -> up [59,128,127,43] acting [59,128,127,43] Internal ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 2

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 22 Mar '21

22 Mar '21

Hi Istvan: Any update for directional sync? I am trying to upgrade Ceph to 15.2.10. Also I have some issues that RGW may crash after I ran data sync init. :( Thanks

1 0

ceph orch daemon add , separate db
by Philip Brown 19 Mar '21

19 Mar '21

I was having difficulty doing this myself, and I came across this semi-recent thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/T4R76XJN2NE… " I've tried adding OSDs with ceph orch daemon add ... but it's pretty limited. ...you can't [have] a separate db device. " Has this been fixed yet? Is it GOING to be fixed? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 1

howto:: emergency shutdown procedure and maintenance
by Adrian Sevcenco 19 Mar '21

19 Mar '21

Hi! What steps/procedures are required for emergency shutdown and for machine maintenance? Thanks a lot! Adrian

2 2