- ceph-users - lists.ceph.io

by Vishwas Bm

Hi, I am a newbie to ceph. I have gone through the ceph docs, we are planning to use rgw for object storage. From the docs, what I have understood is that there are two types of users: 1) ceph storage user 2) radosgw user I am able to create user of both the types. But I am not able to understand how to restrict the rgw user access to a pool. My questions are below: 1) How to restrict the access of a rgw user to a particular pool ? Can this be done using placement groups ? 2) Is it possible to restrict rgw user access to a particular namespace in a pool ? 3) I can understand the flow till he is able to write to a bucket using the .index pool object. But I am not able to understand the flow how the rgw user can write objects in pool. Where can I check the permissions ? *Thanks & Regards,* *Vishwas *

3 years, 11 months

2
2
0 0

Re: Zeroing out rbd image or volume

by Marc Roos

dd if=/dev/zero of=rbd ???? :) but if you have encrypted osd's, what would be the use of this? -----Original Message----- From: huxiaoyu(a)horebdata.cn [mailto:huxiaoyu@horebdata.cn] Sent: 12 May 2020 12:55 To: ceph-users Subject: [ceph-users] Zeroing out rbd image or volume Hi， Ceph folks, Is there a rbd command, or any other way, to zero out rbd images or volume? I would like to write all zero data to an rbd image/volume before remove it. Any comments would be appreciated. best regards, samuel Horebdata AG Switzerland huxiaoyu(a)horebdata.cn _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 11 months

1
0
0 0

Zeroing out rbd image or volume

by huxiaoyu＠horebdata.cn

Hi， Ceph folks, Is there a rbd command, or any other way, to zero out rbd images or volume? I would like to write all zero data to an rbd image/volume before remove it. Any comments would be appreciated. best regards, samuel Horebdata AG Switzerland huxiaoyu(a)horebdata.cn

3 years, 11 months

1
0
0 0

Re: Write Caching to hot tier not working as expected

by Steve Hughes

Thanks Eric. Using your command for SET reported that the OSD may need a restart (which sets it back to default anyway) but the below seems to work: ceph tell osd.24 config set objecter_inflight_op_bytes 1073741824 ceph tell osd.24 config set objecter_inflight_ops 10240 reading back the settings looks right: [root@ceph00 ~]# ceph daemon osd.24 config show | grep objecter "debug_objecter": "0/1", "objecter_completion_locks_per_session": "32", "objecter_debug_inject_relock_delay": "false", "objecter_inflight_op_bytes": "1073741824", "objecter_inflight_ops": "10240", "objecter_inject_no_watch_ping": "false", "objecter_retry_writes_after_first_reply": "false", "objecter_tick_interval": "5.000000", "objecter_timeout": "10.000000", "osd_objecter_finishers": "1", Ive done that for the three OSDs that are in the cache tier. But the performance is unchanged - the writes still spill over to the HDD pool. Still, your idea sounds close - it does feel like something in the cache tier is hitting a limit. Regards, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 9:11 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected Reading and setting them should be pretty easy: READ (Run from the host where OSD <id> is hosted): ceph daemon osd.<id> config show | grep objecter SET (Assuming these can be set in memory): ceph tell osd.<id> injectargs "--objecter-inflight-op-bytes=1073741824" (Change to 1GB/sec throttle) To persist these you should add them to the ceph.conf (I'm not sure what section though - you might have to test this). And yes - the information is sketchy I agree - I don't really have any input here. That's the best I can do for now 😊 Eric -----Original Message----- From: Steve Hughes <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 6:44 AM To: Eric Smith <Eric.Smith(a)vecima.com>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected Thank you Eric. That 'sounds like' exactly my issue. Though I'm surprised to bump into something like that on such a small system and at such low bandwidth. But the information I can find on those parameters is sketchy to say the least. Can you point me at some doco that explains what they do, how to read the current values and how to set them? Cheers, Steve -----Original Message----- From: Eric Smith <Eric.Smith(a)vecima.com> Sent: Monday, 11 May 2020 8:00 PM To: Steve Hughes <steveh(a)scalar.com.au>; ceph-users(a)ceph.io Subject: RE: [ceph-users] Re: Write Caching to hot tier not working as expected It sounds like you might be bumping up against the default objecter_inflight_ops (1024) and/or objecter_inflight_op_bytes (100MB). -----Original Message----- From: steveh(a)scalar.com.au <steveh(a)scalar.com.au> Sent: Monday, May 11, 2020 5:48 AM To: ceph-users(a)ceph.io Subject: [ceph-users] Re: Write Caching to hot tier not working as expected Interestingly, I have found that if I limit the rate at which data is written the tiering behaves as expected. I'm using a robocopy job from a Windows VM to copy large files from my existing storage array to a test Ceph volume. By using the /IPG parameter I can roughly control the rate at which data is written. I've found that if I limit the write rate to around 30MBytes/sec the data all goes to the hot tier, zero data goes to the HDD tier, and the observed write latency is about 5msec. If I go any higher than this I see data being written to the HDDs and the observed write latency goes way up. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io --

3 years, 11 months

1
0
0 0

Cifs slow read speed

by Amudhan P

Hi, I am running a small Ceph Nautilus cluster on Ubuntu 18.04. I am testing cluster to expose cephfs volume in samba v4 share for user to access from windows. When i do test with DD Write (600 MB/s) and md5sum file Read speed is (700 - 800 MB/s) from ceph kernel mount. Same volume i have exposed in samba using "vfs_ceph" and mounted it thru cifs in another ubuntu18.04 as client. Now, when i perform DD write i get speed of 600 MB/s and md5sum of file Read speed is only 65 MB/s. What could be the problem? Any one faced similar issue?

3 years, 11 months

1
0
0 0

Re: Elasticsearch Sync module bug ?

by Cervigni, Luca (Pawsey, Kensington WA)

Thanks for the reply I moved the 6.8.8 and the JSON parsing error seems gone. I am trying to do a simple request though I have very weird results: --- Request URL = https://xxx.xxx.xxx.xxx:6666/test2?query=name%3D%3Dfile-020 {'Host': 'xxx.xxx.xxx', 'Content-length': '0', 'X-Amz-Content-SHA256': 'UNSIGNED-PAYLOAD', 'X-Amz-Date': '20200512T053711Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxx/20200512/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'} Response Response code: 200 <SearchMetadataResponse><Marker></Marker><IsTruncated>false</IsTruncated></SearchMetadataResponse> SearchMetadataResponse: None Marker: None IsTruncated: false --- ________________________________ From: Cervigni, Luca (Pawsey, Kensington WA) <Luca.Cervigni(a)csiro.au> Sent: Monday, 11 May 2020 4:34 PM To: Peter Parker <346415320(a)qq.com>; Luca Cervigni <luca.cervigni(a)pawsey.org.au> Subject: Re: Re:[ceph-users] Elasticsearch Sync module bug ? Thanks Peter for your reply. Could you point me out to the correct settings to put? I tried as well ES 6.8 without success. I just use the default docker container that the ES documentations points out. Could you give me a little more details? Many thanks Regards, Luca Cervigni ________________________________ From: Peter Parker <346415320(a)qq.com> Sent: Monday, 11 May 2020 4:00 PM To: Luca Cervigni <luca.cervigni(a)pawsey.org.au> Subject: Re:[ceph-users] Elasticsearch Sync module bug ? Hi, Luca In my env, ES 7.x is not a valid metadata tier by default for reasons such as index type. but it could be fixed by changing default setting. (I haven't tried) I use ES 6.8 instead and it works fine. ------------------ Original ------------------ From: "Luca Cervigni"<luca.cervigni(a)pawsey.org.au>; Date: Mon, May 11, 2020 12:27 PM To: "ceph-users"<ceph-users(a)ceph.io>; Subject: [ceph-users] Elasticsearch Sync module bug ? Hello, I have a two zones cluster. Default zone + a ES sync zone where I would like to index metadata and do searches via the ES tier gateway. Struggling to get this service running (documentation is bad and very, very minimal) I thought that in the end I made it work. The requests are getting correctly to the ES zone, I can see the payload sent to ES with debug rgw = 20, ES returns OK then an error is trigger, apparently a JSON parsing error taking the ES search and giving it back to the user. I tried with Nautilus, Octopus, and different versions of ES, from 6 to 7. I have no possibilities of downgrades as our production cluster is on nautilus. Here the logs: https://pastebin.com/hnj8YysK I would think this is a bug, but would like your opinion. If not a bug how shall I solve this? I am using a handmade S3 tool to query as boto is not supported anymore, obo just does not work, and boto3 cannot send any arbitrary requests anymore. Also I would like to know if those Sync modules are still supported or not. The documentation is very poor and if they still are in the roadmap, it should be improved. Thanks Cheers, Luca Cervigni Pawsey Supercomputing centre _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 11 months

1
0
0 0

1 pg unknown (from cephfs data pool)

by Marc Roos

I had a 1x replicated fs data test pool, when a osd died I had '1 pg stale+active+clean of cephfs'[1] after a cluster reboot this turned into '1 pg unknown' ceph pg repair did not fix anything (for stale and unknown state) I recreated the pg with: ceph osd force-create-pg pg.id --yes-i-really-mean-it Question now is: Say I had one or two files in this pool/pg, is this still administered in the mds server? Do I need to fix something in the mds? PS. This was just performance testing pool, so at most there could be a few testing images on it, nothing important. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03147.html

3 years, 11 months

1
0
0 0

MDS_CACHE_OVERSIZED warning

by jesper＠krogh.cc

Hi. I have a cluster that has been running for close to 2 years now - pretty much with the same setting, but over the past day I'm seeing this warning. (and the cache seem to keep growing) - Can I figure out which clients is accumulating the inodes? Ceph 12.2.8 - is it ok just to "bump" the memory to say 128GB - any negative sideeffects? jk@ceph-mon1:~$ sudo ceph health detail HEALTH_WARN 1 MDSs report oversized cache; 3 clients failing to respond to cache pressure MDS_CACHE_OVERSIZED 1 MDSs report oversized cache mdsceph-mds1(mds.0): MDS cache is too large (91GB/32GB); 34400070 inodes in use by clients, 3293 stray files Thanks - Jesper

3 years, 11 months

2
1
0 0

Yet another meltdown starting

by Frank Schilder

Hi all, another client-load induced meltdown. It is just starting and I hope we get it under control. This time, its the MGRs failing under the load. It looks like thay don't manage to get their beacons to the mons and are kicked out as unresponsive. However, the processes are fine and up. Its just an enormous load. I'm trying to increase # ceph config set global mon_mgr_beacon_grace 90 but the command doesn't complete. I guess because all the MGRs are out. Is there any way to force the MONs *not* to mark MGRs as unresponsive? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 years, 11 months

2
4
0 0

Recover datas from pg incomplete

by Francois Legrand

Hi, After a major crash in which we lost few osds, we are stucked with incomplete pgs. At first, peering was blocked with peering_blocked_by_history_les_bound. Thus we set osd_find_best_info_ignore_history_les true for all osds involved in the pg and set the primary osd down to force repeering. It worked for one pg which is in a replica 3 pool, but for the 2 others pgs which are in a erasurce coding (3+2) pool, it didn't worked... and the pgs are still incomplete. We know that we will have data lost, but we would like to minimize it and save as much as possible. Also because this pg is part of the data pool of a cephfs filesystem and it seems that files are spread among a lot of pgs and loosing objects in a pg of the datapool means the loss of a huge number of files ! According to https://www.spinics.net/lists/ceph-devel/msg41665.html a way would be to : - stop each osd involved in that pg - export the shards with ceph-objectstore-tool - compare the size of the shards and select the biggest one (alternatively maybe we can also look at the num_objects returned by ceph pg query ?) - Mark it as complete - restart the osd - Wait for recover and finally get rid of the missing objects with ceph pg 10.2 mark_unfound_lost delete But on this other source https://github.com/TheJJ/ceph-cheatsheet/blob/master/README.md or here https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-… it's suggested to remove the other parts (but I am not sure these threads are really related to EC pools). Could you confirm that we could follow this procedure (or correct it or suggests anything else) ? Thanks for your advices. F. PS: Here is a part of the ceph pg 10.2 query return : "state": "incomplete", "snap_trimq": "[]", "snap_trimq_len": 0, "epoch": 434321, "up": [ 78, 105, 90, 4, 41 ], "acting": [ 78, 105, 90, 4, 41 ], "info": { "pgid": "10.2s0", "state": "incomplete", "last_peered": "2020-04-22 09:58:42.505638", "last_became_peered": "2020-04-20 11:06:07.701833", "num_objects": 161314, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 161314, "num_objects_recovered": 1290285, "peer_info": [ "peer": "4(3)", "pgid": "10.2s3", "state": "active+undersized+degraded+remapped+backfilling", "last_peered": "2020-04-25 13:25:12.860435", "last_became_peered": "2020-04-22 10:45:45.520125", "num_objects": 162869, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 85071, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 162869, "num_objects_recovered": 1368082, "peer": "9(2)", "pgid": "10.2s2", "state": "down", "last_peered": "2020-04-25 13:25:12.860435", "last_became_peered": "2020-04-22 10:45:45.520125", "num_objects": 162869, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 162869, "num_objects_recovered": 1368082, "peer": "41(4)", "pgid": "10.2s4", "state": "unknown", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "peer": "46(4)", "pgid": "10.2s4", "state": "down", "last_peered": "2020-04-25 13:25:12.860435", "last_became_peered": "2020-04-22 10:45:45.520125", "num_objects": 162869, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 162869, "num_objects_recovered": 1368082, "peer": "52(3)", "pgid": "10.2s3", "state": "unknown", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "peer": "69(0)", "pgid": "10.2s0", "state": "unknown", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 84063, "num_objects_missing_on_primary": 0, "num_objects_missing": 78807, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 84063, "num_objects_recovered": 0, "peer": "90(2)", "pgid": "10.2s2", "state": "down", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "peer": "105(1)", "pgid": "10.2s1", "state": "incomplete", "last_peered": "0.000000", "last_became_peered": "0.000000", "num_objects": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_objects_recovered": 0, "recovery_state": [ "peering_blocked_by": [] "agent_state": {}

3 years, 11 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users