- ceph-users - lists.ceph.io

disk scheduler for SSD
by George Shuklin 18 Sep '20

18 Sep '20

I start to wonder (again) which scheduler is better for ceph on SSD. My reasoning. None: 1. Reduces latency for requests. The lower latency is, the higher is perceived performance for unbounded workload with fixed queue depth (hello, benchmarks). 2. Causes possible spikes in latency for requests because of the 'unfair' request ordering (hello, deep scrub). Deadline-mq: 1. Reduce size of nr_requests (queue size) to 256 (noop shows me 916???). Make introduce latency. 2. May reduce latency spikes due to different rates for different types of workloads. I'm doing some benchmarks, and they, but of course, gives higher marks for 'none' scheduler. Nevertheless, I believe most of normal workload on Ceph does not utilize it with unbounded rate, so bounded (f.e. app making IO based on external independed events) workload can be hurt by lack of disk scheduler in presence of unbounded workload. Any ideas?

1 0

Problem with manual deep-scrubbing PGs on EC pools
by Osiński Piotr 18 Sep '20

18 Sep '20

Hi, We have a little problem with deep-scrubbing on PGs on EC pool. [root@mon-1 ~]# ceph health detail HEALTH_WARN 1 pgs not deep-scrubbed in time PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time pg 14.d4 not deep-scrubbed since 2020-09-05 20:26:02.696191 [root@mon-1 ~]# ceph pg deep-scrub 14.d4 instructing pg 14.d4s0 on osd.113 to deep-scrub [root@mon-1 ~]# grep deep-scrub /var/log/ceph/ceph.log |grep 14.d4 There is nothing about pg 14.d4. I checked, that pg 14.d4 belongs to pool default.rgw.buckets.data-ec [root@mon-1 ~]# ceph osd pool ls detail |grep default.rgw.buckets.data-ec pool 14 'default.rgw.buckets.data-ec' erasure size 8 min_size 6 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 74563 flags hashpspool stripe_width 4160 application rgw [root@mon-1 ~]# ceph pg ls-by-pool default.rgw.buckets.data-ec |grep 14.d4 14.d4 0 0 0 0 0 0 0 0 active+clean 67m 0'0 74562:10673 [113,40,125,80,16,24,95,32]p113 [113,40,125,80,16,24,95,32]p113 2020- 09-12 14:47:40.603264 2020-09-05 20:26:02.696191 When I try to run a manual scrub or deep-scrub on any PG that belongs to the EC pool it doesn't work. For other PGs in replicated pools, it works fine. Is it possible to manually run pg deep-scrub on an EC pool? Spółki Grupy Wirtualna Polska: Wirtualna Polska Holding Spółka Akcyjna z siedzibą w Warszawie, ul. Żwirki i Wigury 16, 02-092 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 0000407130, kapitał zakładowy: 1 454 218,50 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 521-31-11-513 Wirtualna Polska Media Spółka Akcyjna z siedzibą w Warszawie, ul. Żwirki i Wigury 16, 02-092 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 0000580004, kapitał zakładowy: 320 005 950,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 527-26-45-593 Administratorem udostępnionych danych osobowych jest Wirtualna Polska Media S.A. z siedzibą w Warszawie (dalej „WPM”). WPM przetwarza Twoje dane osobowe, które zostały podane przez Ciebie dobrowolnie w trakcie dotychczasowej współpracy, w związku z zawarciem umowy lub zostały zebrane ze źródeł powszechnie dostępnych, w szczególności: imię i nazwisko, adres email, numer telefonu. Przetwarzamy te dane w celach opisanych w polityce prywatności<https://onas.wp.pl/poufnosc.html>, między innymi w celu realizacji współpracy, realizacji obowiązków przewidzianych prawem, w celach marketingowych WP. Podstawą prawną przetwarzania Twoich danych osobowych w celach marketingowych jest prawnie uzasadniony interes jakim jest m.in. przesyłanie informacji marketingowych o usługach WP, w tym zaproszeń na konferencje branżowe, informacje o publikacjach. Twoje dane możemy przekazywać podmiotom przetwarzającym je na nasze zlecenie oraz podmiotom uprawnionym do uzyskania danych na podstawie obowiązującego prawa. Masz prawo m.in. do żądania dostępu do danych, sprostowania, usunięcia lub ograniczenia ich przetwarzania, jak również prawo do zgłoszenia sprzeciwu w przewidzianych w prawie sytuacjach. Prawa te oraz sposób ich realizacji opisaliśmy w polityce prywatności<https://onas.wp.pl/poufnosc.html>. Tam też znajdziesz informacje jak zakomunikować nam Twoją wolę skorzystania z tych praw.

1 0

RGW multisite replication doesn't start
by Eugen Block 18 Sep '20

18 Sep '20

Hi *, I have 2 virtual one-node-clusters configured for multisite RGW. In the beginning the replication actually worked for some hundred MB or so, and then it stopped. In the meantime I wiped both RGWs twice to make sure the configuration is right (including wiping all pools clean). I don't see any errors in the logs but nothing happens on the secondary site. Both clusters are healthy, RGWs run with https. Uploading data directly to the secondary site also works, so the configuration seems ok to me. These is the current rgw status: ---snip--- primary:~ # radosgw-admin sync status realm c7d5fd30-9c06-46a1-baf4-497f95bf3abc (hamburg) zonegroup 68adec15-aace-403d-bd63-f5182a6437b1 (zg-hamburg) zone 0fb33fa1-8110-4179-ae45-acf5f5f825c5 (z-primary) metadata sync no sync (zone is master) secondary:~ # radosgw-admin sync status 2020-09-17T09:34:59.593+0200 7fdd3e706a40 1 Cannot find zone id=93ece7a6-beef-4f4e-841a-60ba0405f192 (name=z-secondary), switching to local zonegroup configuration realm c7d5fd30-9c06-46a1-baf4-497f95bf3abc (hamburg) zonegroup 68adec15-aace-403d-bd63-f5182a6437b1 (zg-hamburg) zone 93ece7a6-beef-4f4e-841a-60ba0405f192 (z-secondary) metadata sync syncing full sync: 64/64 shards full sync: 3 entries to sync incremental sync: 0/64 shards metadata is behind on 64 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] data sync source: 0fb33fa1-8110-4179-ae45-acf5f5f825c5 (z-primary) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source ---snip--- Since the data was not replicated I ran a 'radosgw-admin metadata sync run --source-zone=z-primary' but it never finishes. If I do the same with data it will show that all shards are behind on data but nothing will happen either. I also don't understand the 'Cannot find zone id=93ece7a6-beef-4f4e-841a-60ba0405f192 (name=z-secondary), switching to local zonegroup configuration' message but this didn't break the replication in the first attempt, so I ignored it. Or is this something I should fix first (if yes, how)? Can anyone point me to what's going on here? I can provide more details if necessary, just let me know. Thank you! Eugen

1 0

Spanning OSDs over two drives
by Liam MacKenzie 18 Sep '20

18 Sep '20

Hi all I have a scenario where I'm upgrading to ceph octopus on hardware that groups its drives in trays which contain 2 devices each. Previously these drives were joined in a software RAID1 and the md devices were used as the OSDs. The logic behind this is that should one of those drives fail, both will need to be removed at the same time due to the design of the machine. For example: https://www.servethehome.com/supermicro-ssg-6047r-e1r72l-72x-35-drive-4u-st… As I understand that using RAID isn't recommended, how would I best deploy my cluster so it's smart enough to group drives according to the trays that they're in? Thanks!

3 2

Introduce flash OSD's to Nautilus installation
by Mathias Lindberg 18 Sep '20

18 Sep '20

Hi, We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup. The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash. We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively . But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating. But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong? I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all. The plan is to then change the crush rule for the exiting pools to the new ones with device classes. Thank you for any pointers or advice. [root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr --- crush_comp 2020-09-17 17:11:37.125310334 +0200 +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 @@ -574,6 +574,7 @@ root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily + id -29 class ssd # weight 2087.180 alg straw2 hash 0 # rjenkins1 @@ -710,5 +711,25 @@ step chooseleaf firstn 0 type host step emit } +rule replicated_ssd { + id 15 + type replicated + min_size 1 + max_size 10 + step take default class ssd + step chooseleaf firstn 0 type host + step emit +} +rule cephyrfs_data_hdd { + id 16 + type erasure + min_size 3 + max_size 12 + step set_chooseleaf_tries 5 + step set_choose_tries 100 + step take default class hdd + step chooseleaf indep 0 type host + step emit +} # end crush map [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 1 had 0/6144 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) rule 14 had 0/10240 mismatched mappings (0) maps appear equivalent Regards, Mathias Lindberg Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin(a)chalmers.se

3 3

Nautilus Scrub and deep-Scrub execution order
by Johannes L 17 Sep '20

17 Sep '20

Hello Ceph-Users after upgrading one of our clusters to Nautilus we noticed the x pgs not scrubbed/deep-scrubbed in time warnings. Through some digging we found out that it seems like the scrubbing takes place at random and doesn't take the age of the last scrub/deep-scrub into consideration. I dumped the time of the last scrub with a 90 min gap in between: ceph pg dump | grep active | awk '{print $22}' | sort | uniq -c dumped all 2434 2020-08-30 5935 2020-08-31 1782 2020-09-01 2 2020-09-02 2 2020-09-03 5 2020-09-06 3 2020-09-08 5 2020-09-09 17 2020-09-10 259 2020-09-12 26672 2020-09-13 12036 2020-09-14 dumped all 2434 2020-08-30 5933 2020-08-31 1782 2020-09-01 2 2020-09-02 2 2020-09-03 5 2020-09-06 3 2020-09-08 5 2020-09-09 17 2020-09-10 51 2020-09-12 24862 2020-09-13 14056 2020-09-14 It is pretty obvious that the PGs that have been scrubbed a day ago have been scrubbed again for some reason while ones that are 2 weeks old are basically left untouched. One way we are currently dealing with this issue is setting the osd_scrub_min_interval to 72h to force the cluster to scrub the older PGs. This can't be intentional. Has anyone else seen this behavior? Kind regards Johannes

4 4

rbd map on octopus from luminous client
by Marc Boisis 17 Sep '20

17 Sep '20

Hi, I had to map a rbd from an ubuntu Trusty luminous client on an octopus cluster. client dmesg : feature set mismatch, my 4a042a42 < server's 100004a042a42, missing 1000000000000 I downgrade my osd tunable to bobtail but it still doesn't work ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 0, "chooseleaf_stable": 0, "straw_calc_version": 1, "allowed_bucket_algs": 22, "profile": "bobtail", "optimal_tunables": 0, "legacy_tunables": 0, "minimum_required_version": "hammer", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 0, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 0, "has_v5_rules": 0 } Thanks for your help. Marc

2 2

Disk consume for CephFS
by fotofors＠gmail.com 17 Sep '20

17 Sep '20

Hello. I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB of files.I created the filesystem and started to copy files via rsync. However, I've had to stop the process, because Ceph shows me that the new size of the folder is almost 6TB. I double checked the replicated size and it is 2. I double checked the rsync options and I didn't copy the files followed by symlinks. How would it be possible to explain the extreme difference between the size of the original folder and CephFS?

4 4

Migration to ceph.readthedocs.io underway
by Neha Ojha 17 Sep '20

17 Sep '20

Hi everyone, We are in the process of migrating from docs.ceph.com to ceph.readthedocs.io. We enabled it in https://github.com/ceph/ceph/pull/34499 and will now be using it by default. Why? - The search feature in ceph.readthedocs.io is much better than docs.ceph.com and allows you to search multiple strings. - RTD provides an in-built version switching feature which we plan to use in future. What does it mean to you? - Some broken links are expected during this migration. Things like ceph API documentation need special handling (example: https://docs.ceph.com/en/latest/rados/api/) and are expected to be broken temporarily. - Much better Ceph documentation experience once the migration is done. Thanks for your patience! Cheers, Neha

4 5

Misunderstanding in Facebook? Dial Facebook Customer Service Toll Free Number to address rep.
by mary smith 17 Sep '20

17 Sep '20

Every so often there might be a mistake in the inbox and that may dishearten you to see the sends. Considering, you can take a stab at bracing the page. On the off chance that that doesn't work, by then you can find maintain from the client care pack by dialing the Facebook Customer Service Toll Free Number and keeping an eye on the rep or you can cycle to take help from the tech help regions. https://www.customercare-email.com/facebook-customer-service.html

1 0