- ceph-users - lists.ceph.io

Ceph Apply/Commit vs Read/Write Op Latency
by John Petrini 12 May '20

12 May '20

Hello, I was hoping someone could clear up the difference between these metrics. In filestore the difference between Apply and Commit Latency was pretty clear and these metrics gave a good representation of how the cluster was performing. High commit usually meant our journals were performing poorly while high apply pointed to an OSD issue. With bluestore Apply & Commit are now tied to the same metric and it's not as clear to me what that metric is. In addition new metrics such as Read and Write Op Latency have been added. I'm led to believe that these are similar to what Apply Latency used to represent but is that actually the case? If anyone who has a better understanding of this than I do can enlighten me I'd appreciate it! Thanks, John

1 1

Add lvm in cephadm
by Simon Sutter 12 May '20

12 May '20

Hello Everyone, The new cephadm is giving me a headache. I'm setting up a new testenvironment, where I have to use lvm partitions, because I don't have more Hardware. I could't find any information about the compatibility of existing lvm partitions and cephadm/octopus. I tried the old method, where I had to use ceph-volume to use an existing lvm:

2 3

rgw user access questions
by Vishwas Bm 12 May '20

12 May '20

Hi, I am a newbie to ceph. I have gone through the ceph docs, we are planning to use rgw for object storage. From the docs, what I have understood is that there are two types of users: 1) ceph storage user 2) radosgw user I am able to create user of both the types. But I am not able to understand how to restrict the rgw user access to a pool. My questions are below: 1) How to restrict the access of a rgw user to a particular pool ? Can this be done using placement groups ? 2) Is it possible to restrict rgw user access to a particular namespace in a pool ? 3) I can understand the flow till he is able to write to a bucket using the .index pool object. But I am not able to understand the flow how the rgw user can write objects in pool. Where can I check the permissions ? *Thanks & Regards,* *Vishwas *

2 2

Re: Zeroing out rbd image or volume
by Marc Roos 12 May '20

12 May '20

dd if=/dev/zero of=rbd ???? :) but if you have encrypted osd's, what would be the use of this? -----Original Message----- From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] Sent: 12 May 2020 12:55 To: ceph-users Subject: [ceph-users] Zeroing out rbd image or volume Hi， Ceph folks, Is there a rbd command, or any other way, to zero out rbd images or volume? I would like to write all zero data to an rbd image/volume before remove it. Any comments would be appreciated. best regards, samuel Horebdata AG Switzerland huxiaoyu(a)horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Zeroing out rbd image or volume
by huxiaoyu＠horebdata.cn 12 May '20

12 May '20

Hi， Ceph folks, Is there a rbd command, or any other way, to zero out rbd images or volume? I would like to write all zero data to an rbd image/volume before remove it. Any comments would be appreciated. best regards, samuel Horebdata AG Switzerland huxiaoyu(a)horebdata.cn

1 0

Re: Write Caching to hot tier not working as expected
by Steve Hughes 12 May '20

12 May '20

Thanks Eric. Using your command for SET reported that the OSD may need a restart (which sets it back to default anyway) but the below seems to work: ceph tell osd.24 config set objecter_inflight_op_bytes 1073741824 ceph tell osd.24 config set objecter_inflight_ops 10240 reading back the settings looks right: [root@ceph00 ~]# ceph daemon osd.24 config show | grep objecter "debug_objecter": "0/1", "objecter_completion_locks_per_session": "32", "objecter_debug_inject_relock_delay": "false", "objecter_inflight_op_bytes": "1073741824", "objecter_inflight_ops": "10240", "objecter_inject_no_watch_ping": "false", "objecter_retry_writes_after_first_reply": "false", "objecter_tick_interval": "5.000000", "objecter_timeout": "10.000000", "osd_objecter_finishers": "1", Ive done that for the three OSDs that are in the cache tier. But the performance is unchanged - the writes still spill over to the HDD pool. Still, your idea sounds close - it does feel like something in the cache tier is hitting a limit. Regards, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 9:11 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected Reading and setting them should be pretty easy: READ (Run from the host where OSD <id> is hosted): ceph daemon osd.<id> config show | grep objecter SET (Assuming these can be set in memory): ceph tell osd.<id> injectargs "--objecter-inflight-op-bytes=1073741824" (Change to 1GB/sec throttle) To persist these you should add them to the ceph.conf (I'm not sure what section though - you might have to test this). And yes - the information is sketchy I agree - I don't really have any input here. That's the best I can do for now 😊 Eric -----Original Message----- From: Steve Hughes <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 6:44 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected Thank you Eric. That 'sounds like' exactly my issue. Though I'm surprised to bump into something like that on such a small system and at such low bandwidth. But the information I can find on those parameters is sketchy to say the least. Can you point me at some doco that explains what they do, how to read the current values and how to set them? Cheers, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 8:00 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected It sounds like you might be bumping up against the default objecter_inflight_ops (1024) and/or objecter_inflight_op_bytes (100MB). -----Original Message----- From: steveh(a)scalar.com.au <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 5:48 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Write Caching to hot tier not working as expected Interestingly, I have found that if I limit the rate at which data is written the tiering behaves as expected. I'm using a robocopy job from a Windows VM to copy large files from my existing storage array to a test Ceph volume. By using the /IPG parameter I can roughly control the rate at which data is written. I've found that if I limit the write rate to around 30MBytes/sec the data all goes to the hot tier, zero data goes to the HDD tier, and the observed write latency is about 5msec. If I go any higher than this I see data being written to the HDDs and the observed write latency goes way up. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io --

1 0

Cifs slow read speed
by Amudhan P 12 May '20

12 May '20

Hi, I am running a small Ceph Nautilus cluster on Ubuntu 18.04. I am testing cluster to expose cephfs volume in samba v4 share for user to access from windows. When i do test with DD Write (600 MB/s) and md5sum file Read speed is (700 - 800 MB/s) from ceph kernel mount. Same volume i have exposed in samba using "vfs_ceph" and mounted it thru cifs in another ubuntu18.04 as client. Now, when i perform DD write i get speed of 600 MB/s and md5sum of file Read speed is only 65 MB/s. What could be the problem? Any one faced similar issue?

1 0

Re: Elasticsearch Sync module bug ?
by Cervigni, Luca (Pawsey, Kensington WA) 12 May '20

12 May '20

Thanks for the reply I moved the 6.8.8 and the JSON parsing error seems gone. I am trying to do a simple request though I have very weird results: --- Request URL = https://xxx.xxx.xxx.xxx:6666/test2?query=name%3D%3Dfile-020 {'Host': 'xxx.xxx.xxx', 'Content-length': '0', 'X-Amz-Content-SHA256': 'UNSIGNED-PAYLOAD', 'X-Amz-Date': '20200512T053711Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxx/20200512/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'} Response Response code: 200 <SearchMetadataResponse><Marker></Marker><IsTruncated>false</IsTruncated></SearchMetadataResponse> SearchMetadataResponse: None Marker: None IsTruncated: false --- ________________________________ From: Cervigni, Luca (Pawsey, Kensington WA) <Luca.Cervigni(a)csiro.au> Sent: Monday, 11 May 2020 4:34 PM To: Peter Parker <346415320(a)qq.com>; Luca Cervigni <luca.cervigni(a)pawsey.org.au> Subject: Re: Re:[ceph-users] Elasticsearch Sync module bug ? Thanks Peter for your reply. Could you point me out to the correct settings to put? I tried as well ES 6.8 without success. I just use the default docker container that the ES documentations points out. Could you give me a little more details? Many thanks Regards, Luca Cervigni ________________________________ From: Peter Parker <346415320(a)qq.com> Sent: Monday, 11 May 2020 4:00 PM To: Luca Cervigni <luca.cervigni(a)pawsey.org.au> Subject: Re:[ceph-users] Elasticsearch Sync module bug ? Hi, Luca In my env, ES 7.x is not a valid metadata tier by default for reasons such as index type. but it could be fixed by changing default setting. (I haven't tried) I use ES 6.8 instead and it works fine. ------------------ Original ------------------ From: "Luca Cervigni"<luca.cervigni(a)pawsey.org.au>; Date: Mon, May 11, 2020 12:27 PM To: "ceph-users"<ceph-users(a)ceph.io>; Subject: [ceph-users] Elasticsearch Sync module bug ? Hello, I have a two zones cluster. Default zone + a ES sync zone where I would like to index metadata and do searches via the ES tier gateway. Struggling to get this service running (documentation is bad and very, very minimal) I thought that in the end I made it work. The requests are getting correctly to the ES zone, I can see the payload sent to ES with debug rgw = 20, ES returns OK then an error is trigger, apparently a JSON parsing error taking the ES search and giving it back to the user. I tried with Nautilus, Octopus, and different versions of ES, from 6 to 7. I have no possibilities of downgrades as our production cluster is on nautilus. Here the logs: https://pastebin.com/hnj8YysK I would think this is a bug, but would like your opinion. If not a bug how shall I solve this? I am using a handmade S3 tool to query as boto is not supported anymore, obo just does not work, and boto3 cannot send any arbitrary requests anymore. Also I would like to know if those Sync modules are still supported or not. The documentation is very poor and if they still are in the roadmap, it should be improved. Thanks Cheers, Luca Cervigni Pawsey Supercomputing centre _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

1 pg unknown (from cephfs data pool)
by Marc Roos 11 May '20

11 May '20

I had a 1x replicated fs data test pool, when a osd died I had '1 pg stale+active+clean of cephfs'[1] after a cluster reboot this turned into '1 pg unknown' ceph pg repair did not fix anything (for stale and unknown state) I recreated the pg with: ceph osd force-create-pg pg.id --yes-i-really-mean-it Question now is: Say I had one or two files in this pool/pg, is this still administered in the mds server? Do I need to fix something in the mds? PS. This was just performance testing pool, so at most there could be a few testing images on it, nothing important. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03147.html

1 0

MDS_CACHE_OVERSIZED warning
by jesper＠krogh.cc 11 May '20

11 May '20

Hi. I have a cluster that has been running for close to 2 years now - pretty much with the same setting, but over the past day I'm seeing this warning. (and the cache seem to keep growing) - Can I figure out which clients is accumulating the inodes? Ceph 12.2.8 - is it ok just to "bump" the memory to say 128GB - any negative sideeffects? jk@ceph-mon1:~$ sudo ceph health detail HEALTH_WARN 1 MDSs report oversized cache; 3 clients failing to respond to cache pressure MDS_CACHE_OVERSIZED 1 MDSs report oversized cache mdsceph-mds1(mds.0): MDS cache is too large (91GB/32GB); 34400070 inodes in use by clients, 3293 stray files Thanks - Jesper

2 1