September 2020 - ceph-users

RuntimeError: Unable check if OSD id exists
by Marc Roos 18 Sep '20

18 Sep '20

I have still ceph-disk created osd's in nautilus. Thought about using this ceph-volume, but looks like this manual for replacing ceph-disk[1] is not complete. Getting already this error RuntimeError: Unable check if OSD id exists: [1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/#rados-repl…

1 0

Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
by Danni Setiawan 18 Sep '20

18 Sep '20

Hi all, I'm trying to find performance penalty with OSD HDD when using WAL/DB in faster device (SSD/NVMe) vs WAL/DB in same device (HDD) for different workload (RBD, RGW with index bucket in SSD pool, and CephFS with metadata in SSD pool). I want to know if giving up disk slot for WAL/DB device is worth vs adding more OSD. Unfortunately I cannot find the benchmark for these kind workload. Has anyone ever done this benchmark? Thank you.

8 10

Using cephadm shell/ceph-volume
by tri＠postix.net 18 Sep '20

18 Sep '20

Hi all, I'm having problem creating an osd using ceph-volume (by the way of cephadm). This is on an octopus installation with cephadm. So I use "cephadm shell" and then "ceph-volume" but got the following error: root@furry:/var/lib# ceph-volume lvm prepare --data /dev/sda --block.db /dev/vg/sda.db --dmcrypt Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 29d95564-733d-4f2a-a2c8-1bb9ceb5a14b stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id if I pass the cluster ID, and use the correct key ring (using the client.admin keyring), I got a bit further root@furry:/var/lib# ceph-volume --cluster c258000c-f3e4-11ea-9ebe-c3c75e8e9028 lvm prepare --data /dev/sda --block.db /dev/vg/sda.db --dmcrypt Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster c258000c-f3e4-11ea-9ebe-c3c75e8e9028 --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/c258000c-f3e4-11ea-9ebe-c3c75e8e9028.keyring -i - osd new aa13c362-c9cf-4d03-9a86-d6118fbc312c stderr: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)',) --> RuntimeError: Unable to create a new OSD id Any idea on how to get pass these errors? Thanks. --Tri Hoang

2 1

It is an advantage to use assignment help in Australia
by john seena 18 Sep '20

18 Sep '20

Do you want to connect with Australian assignment helpers to feel the low stress of submission? Are you seeking the best source of help for composing your academic documents? In this context, use Assignment Help Services and get completed papers without any delay. As I can understand that, whenever you have issues in writing your assignments, it gets hard for you to write flawless papers. When you are busy with lots of activities, you can’t focus on assignment writing and collecting requisite information for it. So, if you require a reliable way of finishing your homework, choosing the assistance of professionals is an advantage. You can manage your effort and save your time using the services of online academic writing. I can say so because I already used these services when I was studying Australia. You don’t have to think a lot because using online assignment help services is the best choice for completing your work without any issue. Don’t face any delay if you know about online academic writing service in Australia. https://www.greatassignmenthelp.com/au/

1 0

September Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 18 Sep '20

18 Sep '20

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday September 23rd. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200923 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: September 23, 2020 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0

disk scheduler for SSD
by George Shuklin 18 Sep '20

18 Sep '20

I start to wonder (again) which scheduler is better for ceph on SSD. My reasoning. None: 1. Reduces latency for requests. The lower latency is, the higher is perceived performance for unbounded workload with fixed queue depth (hello, benchmarks). 2. Causes possible spikes in latency for requests because of the 'unfair' request ordering (hello, deep scrub). Deadline-mq: 1. Reduce size of nr_requests (queue size) to 256 (noop shows me 916???). Make introduce latency. 2. May reduce latency spikes due to different rates for different types of workloads. I'm doing some benchmarks, and they, but of course, gives higher marks for 'none' scheduler. Nevertheless, I believe most of normal workload on Ceph does not utilize it with unbounded rate, so bounded (f.e. app making IO based on external independed events) workload can be hurt by lack of disk scheduler in presence of unbounded workload. Any ideas?

1 0

Problem with manual deep-scrubbing PGs on EC pools
by Osiński Piotr 18 Sep '20

18 Sep '20

Hi, We have a little problem with deep-scrubbing on PGs on EC pool. [root@mon-1 ~]# ceph health detail HEALTH_WARN 1 pgs not deep-scrubbed in time PG_NOT_DEEP_SCRUBBED 1 pgs not deep-scrubbed in time pg 14.d4 not deep-scrubbed since 2020-09-05 20:26:02.696191 [root@mon-1 ~]# ceph pg deep-scrub 14.d4 instructing pg 14.d4s0 on osd.113 to deep-scrub [root@mon-1 ~]# grep deep-scrub /var/log/ceph/ceph.log |grep 14.d4 There is nothing about pg 14.d4. I checked, that pg 14.d4 belongs to pool default.rgw.buckets.data-ec [root@mon-1 ~]# ceph osd pool ls detail |grep default.rgw.buckets.data-ec pool 14 'default.rgw.buckets.data-ec' erasure size 8 min_size 6 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 74563 flags hashpspool stripe_width 4160 application rgw [root@mon-1 ~]# ceph pg ls-by-pool default.rgw.buckets.data-ec |grep 14.d4 14.d4 0 0 0 0 0 0 0 0 active+clean 67m 0'0 74562:10673 [113,40,125,80,16,24,95,32]p113 [113,40,125,80,16,24,95,32]p113 2020- 09-12 14:47:40.603264 2020-09-05 20:26:02.696191 When I try to run a manual scrub or deep-scrub on any PG that belongs to the EC pool it doesn't work. For other PGs in replicated pools, it works fine. Is it possible to manually run pg deep-scrub on an EC pool? Spółki Grupy Wirtualna Polska: Wirtualna Polska Holding Spółka Akcyjna z siedzibą w Warszawie, ul. Żwirki i Wigury 16, 02-092 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 0000407130, kapitał zakładowy: 1 454 218,50 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 521-31-11-513 Wirtualna Polska Media Spółka Akcyjna z siedzibą w Warszawie, ul. Żwirki i Wigury 16, 02-092 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 0000580004, kapitał zakładowy: 320 005 950,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 527-26-45-593 Administratorem udostępnionych danych osobowych jest Wirtualna Polska Media S.A. z siedzibą w Warszawie (dalej „WPM”). WPM przetwarza Twoje dane osobowe, które zostały podane przez Ciebie dobrowolnie w trakcie dotychczasowej współpracy, w związku z zawarciem umowy lub zostały zebrane ze źródeł powszechnie dostępnych, w szczególności: imię i nazwisko, adres email, numer telefonu. Przetwarzamy te dane w celach opisanych w polityce prywatności<https://onas.wp.pl/poufnosc.html>, między innymi w celu realizacji współpracy, realizacji obowiązków przewidzianych prawem, w celach marketingowych WP. Podstawą prawną przetwarzania Twoich danych osobowych w celach marketingowych jest prawnie uzasadniony interes jakim jest m.in. przesyłanie informacji marketingowych o usługach WP, w tym zaproszeń na konferencje branżowe, informacje o publikacjach. Twoje dane możemy przekazywać podmiotom przetwarzającym je na nasze zlecenie oraz podmiotom uprawnionym do uzyskania danych na podstawie obowiązującego prawa. Masz prawo m.in. do żądania dostępu do danych, sprostowania, usunięcia lub ograniczenia ich przetwarzania, jak również prawo do zgłoszenia sprzeciwu w przewidzianych w prawie sytuacjach. Prawa te oraz sposób ich realizacji opisaliśmy w polityce prywatności<https://onas.wp.pl/poufnosc.html>. Tam też znajdziesz informacje jak zakomunikować nam Twoją wolę skorzystania z tych praw.

1 0

RGW multisite replication doesn't start
by Eugen Block 18 Sep '20

18 Sep '20

Hi *, I have 2 virtual one-node-clusters configured for multisite RGW. In the beginning the replication actually worked for some hundred MB or so, and then it stopped. In the meantime I wiped both RGWs twice to make sure the configuration is right (including wiping all pools clean). I don't see any errors in the logs but nothing happens on the secondary site. Both clusters are healthy, RGWs run with https. Uploading data directly to the secondary site also works, so the configuration seems ok to me. These is the current rgw status: ---snip--- primary:~ # radosgw-admin sync status realm c7d5fd30-9c06-46a1-baf4-497f95bf3abc (hamburg) zonegroup 68adec15-aace-403d-bd63-f5182a6437b1 (zg-hamburg) zone 0fb33fa1-8110-4179-ae45-acf5f5f825c5 (z-primary) metadata sync no sync (zone is master) secondary:~ # radosgw-admin sync status 2020-09-17T09:34:59.593+0200 7fdd3e706a40 1 Cannot find zone id=93ece7a6-beef-4f4e-841a-60ba0405f192 (name=z-secondary), switching to local zonegroup configuration realm c7d5fd30-9c06-46a1-baf4-497f95bf3abc (hamburg) zonegroup 68adec15-aace-403d-bd63-f5182a6437b1 (zg-hamburg) zone 93ece7a6-beef-4f4e-841a-60ba0405f192 (z-secondary) metadata sync syncing full sync: 64/64 shards full sync: 3 entries to sync incremental sync: 0/64 shards metadata is behind on 64 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] data sync source: 0fb33fa1-8110-4179-ae45-acf5f5f825c5 (z-primary) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source ---snip--- Since the data was not replicated I ran a 'radosgw-admin metadata sync run --source-zone=z-primary' but it never finishes. If I do the same with data it will show that all shards are behind on data but nothing will happen either. I also don't understand the 'Cannot find zone id=93ece7a6-beef-4f4e-841a-60ba0405f192 (name=z-secondary), switching to local zonegroup configuration' message but this didn't break the replication in the first attempt, so I ignored it. Or is this something I should fix first (if yes, how)? Can anyone point me to what's going on here? I can provide more details if necessary, just let me know. Thank you! Eugen

1 0

Spanning OSDs over two drives
by Liam MacKenzie 18 Sep '20

18 Sep '20

Hi all I have a scenario where I'm upgrading to ceph octopus on hardware that groups its drives in trays which contain 2 devices each. Previously these drives were joined in a software RAID1 and the md devices were used as the OSDs. The logic behind this is that should one of those drives fail, both will need to be removed at the same time due to the design of the machine. For example: https://www.servethehome.com/supermicro-ssg-6047r-e1r72l-72x-35-drive-4u-st… As I understand that using RAID isn't recommended, how would I best deploy my cluster so it's smart enough to group drives according to the trays that they're in? Thanks!

3 2

Introduce flash OSD's to Nautilus installation
by Mathias Lindberg 17 Sep '20

17 Sep '20

Hi, We have a 1.2PB Nautilus installation primarily using CephFS for our HPC-resources. Our OSD’s have spinning disks and NvME devices for WAL and DB in an LVM-setup. The CephFS metadata pool resides on spinning disks, and I wonder if there is any point from a performance perspective to put that on flash? Trying to google that does not provide for a single straight answer, some say its is heavily cached on the MDS and does not benefit that much from flash while others argue it is significantly faster putting it on flash. We would regardless like to introduce flash OSD’s. In order to do that and not have data moving on to the ssd’s OSD’s we add we need add crush map rules that place data on ssd and hdd exclusively . But from reading previous posts to the list adding rules like that would trigger PG’s to start migrating. But when I (reluctantly) manually edit the crush map, adding a new class ssd (using a previously unused id) and adding the new rules I need crushtool seems to think no data vill shuffle and that maps are equivalent. Is it that simple, or am i doing it wrong? I would have preferred to have used some sort of tool like the crushtool reclassify (if applicable) to make these changes, but I can’t get the hang of that at all. The plan is to then change the crush rule for the exiting pools to the new ones with device classes. Thank you for any pointers or advice. [root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr --- crush_comp 2020-09-17 17:11:37.125310334 +0200 +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 @@ -574,6 +574,7 @@ root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily + id -29 class ssd # weight 2087.180 alg straw2 hash 0 # rjenkins1 @@ -710,5 +711,25 @@ step chooseleaf firstn 0 type host step emit } +rule replicated_ssd { + id 15 + type replicated + min_size 1 + max_size 10 + step take default class ssd + step chooseleaf firstn 0 type host + step emit +} +rule cephyrfs_data_hdd { + id 16 + type erasure + min_size 3 + max_size 12 + step set_chooseleaf_tries 5 + step set_choose_tries 100 + step take default class hdd + step chooseleaf indep 0 type host + step emit +} # end crush map [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare crush_comp_corr.c rule 0 had 0/10240 mismatched mappings (0) rule 1 had 0/6144 mismatched mappings (0) rule 6 had 0/10240 mismatched mappings (0) rule 7 had 0/4096 mismatched mappings (0) rule 8 had 0/4096 mismatched mappings (0) rule 9 had 0/4096 mismatched mappings (0) rule 10 had 0/4096 mismatched mappings (0) rule 11 had 0/4096 mismatched mappings (0) rule 12 had 0/4096 mismatched mappings (0) rule 13 had 0/4096 mismatched mappings (0) rule 14 had 0/10240 mismatched mappings (0) maps appear equivalent Regards, Mathias Lindberg Tel: +46 (0)31 7723059 Mob: +46 (0)723 526107 Mathias Lindberg mathlin(a)chalmers.se

3 3

Nautilus Scrub and deep-Scrub execution order
by Johannes L 17 Sep '20

17 Sep '20

Hello Ceph-Users after upgrading one of our clusters to Nautilus we noticed the x pgs not scrubbed/deep-scrubbed in time warnings. Through some digging we found out that it seems like the scrubbing takes place at random and doesn't take the age of the last scrub/deep-scrub into consideration. I dumped the time of the last scrub with a 90 min gap in between: ceph pg dump | grep active | awk '{print $22}' | sort | uniq -c dumped all 2434 2020-08-30 5935 2020-08-31 1782 2020-09-01 2 2020-09-02 2 2020-09-03 5 2020-09-06 3 2020-09-08 5 2020-09-09 17 2020-09-10 259 2020-09-12 26672 2020-09-13 12036 2020-09-14 dumped all 2434 2020-08-30 5933 2020-08-31 1782 2020-09-01 2 2020-09-02 2 2020-09-03 5 2020-09-06 3 2020-09-08 5 2020-09-09 17 2020-09-10 51 2020-09-12 24862 2020-09-13 14056 2020-09-14 It is pretty obvious that the PGs that have been scrubbed a day ago have been scrubbed again for some reason while ones that are 2 weeks old are basically left untouched. One way we are currently dealing with this issue is setting the osd_scrub_min_interval to 72h to force the cluster to scrub the older PGs. This can't be intentional. Has anyone else seen this behavior? Kind regards Johannes

4 4

rbd map on octopus from luminous client
by Marc Boisis 17 Sep '20

17 Sep '20

Hi, I had to map a rbd from an ubuntu Trusty luminous client on an octopus cluster. client dmesg : feature set mismatch, my 4a042a42 < server's 100004a042a42, missing 1000000000000 I downgrade my osd tunable to bobtail but it still doesn't work ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 0, "chooseleaf_stable": 0, "straw_calc_version": 1, "allowed_bucket_algs": 22, "profile": "bobtail", "optimal_tunables": 0, "legacy_tunables": 0, "minimum_required_version": "hammer", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 0, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 0, "has_v5_rules": 0 } Thanks for your help. Marc

2 2

Disk consume for CephFS
by fotofors＠gmail.com 17 Sep '20

17 Sep '20

Hello. I'm using the Nautilus Ceph version for some huge folder with approximately 1.7TB of files.I created the filesystem and started to copy files via rsync. However, I've had to stop the process, because Ceph shows me that the new size of the folder is almost 6TB. I double checked the replicated size and it is 2. I double checked the rsync options and I didn't copy the files followed by symlinks. How would it be possible to explain the extreme difference between the size of the original folder and CephFS?

4 4

Migration to ceph.readthedocs.io underway
by Neha Ojha 17 Sep '20

17 Sep '20

Hi everyone, We are in the process of migrating from docs.ceph.com to ceph.readthedocs.io. We enabled it in https://github.com/ceph/ceph/pull/34499 and will now be using it by default. Why? - The search feature in ceph.readthedocs.io is much better than docs.ceph.com and allows you to search multiple strings. - RTD provides an in-built version switching feature which we plan to use in future. What does it mean to you? - Some broken links are expected during this migration. Things like ceph API documentation need special handling (example: https://docs.ceph.com/en/latest/rados/api/) and are expected to be broken temporarily. - Much better Ceph documentation experience once the migration is done. Thanks for your patience! Cheers, Neha

4 5

Misunderstanding in Facebook? Dial Facebook Customer Service Toll Free Number to address rep.
by mary smith 17 Sep '20

17 Sep '20

Every so often there might be a mistake in the inbox and that may dishearten you to see the sends. Considering, you can take a stab at bracing the page. On the off chance that that doesn't work, by then you can find maintain from the client care pack by dialing the Facebook Customer Service Toll Free Number and keeping an eye on the rep or you can cycle to take help from the tech help regions. https://www.customercare-email.com/facebook-customer-service.html

1 0

Problem in printhead causing Epson Error Code 0x97? Get to customer care for help.
by mary smith 17 Sep '20

17 Sep '20

Sometimes the printhead issue can be a problem and result in Epson Error Code 0x97. Therefore, there can be problems from time to time and that will cause an issue. To deal with the error, you can take assistance from the tech videos or consultancies. In addition to that, you can also reach customer care for help. https://www.epsonprintersupportpro.net/epson-error-code-0x97/

1 0

vfs_ceph for CentOS 8
by Frank Schilder 17 Sep '20

17 Sep '20

Hi all, we are setting up a SAMBA share and would like to use the vfs_ceph module. Unfortunately, it seems not to be part of the common SAMBA packages on CentOS 8. Does anyone know how to install vfs_ceph? The SAMBA version on CentOS 8 is samba-4.11.2-13 and the documentation says the module is part of it. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

Re: Migration to ceph.readthedocs.io underway
by Neha Ojha 16 Sep '20

16 Sep '20

On Wed, Sep 16, 2020 at 11:08 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > - In the future you will not be able to read the docs if you have an > adblocker(?) not aware of anything of this sort > > > > -----Original Message----- > To: dev; ceph-users > Cc: Kefu Chai > Subject: [ceph-users] Migration to ceph.readthedocs.io underway > > Hi everyone, > > We are in the process of migrating from docs.ceph.com to > ceph.readthedocs.io. We enabled it in > https://github.com/ceph/ceph/pull/34499 and will now be using it by > default. > > Why? > > - The search feature in ceph.readthedocs.io is much better than > docs.ceph.com and allows you to search multiple strings. > - RTD provides an in-built version switching feature which we plan to > use in future. > > What does it mean to you? > > - Some broken links are expected during this migration. Things like ceph > API documentation need special handling (example: > https://docs.ceph.com/en/latest/rados/api/) and are expected to be > broken temporarily. > > - Much better Ceph documentation experience once the migration is done. > > Thanks for your patience! > > Cheers, > Neha > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

Re: Migration to ceph.readthedocs.io underway
by Sasha Litvak 16 Sep '20

16 Sep '20

I wonder if this new system allows me to choose Ceph versions. I see the v:latest in the right bottom corner but it seems to be the only choice so far. On Wed, Sep 16, 2020 at 12:31 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > - In the future you will not be able to read the docs if you have an > adblocker(?) > > > > -----Original Message----- > To: dev; ceph-users > Cc: Kefu Chai > Subject: [ceph-users] Migration to ceph.readthedocs.io underway > > Hi everyone, > > We are in the process of migrating from docs.ceph.com to > ceph.readthedocs.io. We enabled it in > https://github.com/ceph/ceph/pull/34499 and will now be using it by > default. > > Why? > > - The search feature in ceph.readthedocs.io is much better than > docs.ceph.com and allows you to search multiple strings. > - RTD provides an in-built version switching feature which we plan to > use in future. > > What does it mean to you? > > - Some broken links are expected during this migration. Things like ceph > API documentation need special handling (example: > https://docs.ceph.com/en/latest/rados/api/) and are expected to be > broken temporarily. > > - Much better Ceph documentation experience once the migration is done. > > Thanks for your patience! > > Cheers, > Neha > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

v15.2.5 octopus released
by Abhishek Lekshmanan 16 Sep '20

16 Sep '20

This is the fifth backport release of the Ceph Octopus stable release series. This release brings a range of fixes across all components. We recommend that all Octopus users upgrade to this release. Notable Changes --------------- * CephFS: Automatic static subtree partitioning policies may now be configured using the new distributed and random ephemeral pinning extended attributes on directories. See the documentation for more information: https://docs.ceph.com/docs/master/cephfs/multimds/ * Monitors now have a config option `mon_osd_warn_num_repaired`, 10 by default. If any OSD has repaired more than this many I/O errors in stored data a `OSD_TOO_MANY_REPAIRS` health warning is generated. * Now when noscrub and/or no deep-scrub flags are set globally or per pool, scheduled scrubs of the type disabled will be aborted. All user initiated scrubs are NOT interrupted. * Fix an issue with osdmaps not being trimmed in a healthy cluster ( issue#47297, pr#36981) For the detailed changelog please refer to the blog entry at https://ceph.io/releases/v15-2-5-octopus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 2c93eff00150f0cc5f106a559557a58d3d7b6f1f -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)

1 0

ceph pgs inconsistent, always the same checksum
by David Orman 16 Sep '20

16 Sep '20

Hi, I've got a ceph cluster, 7 nodes, 168 OSDs, with 96G of ram on each server. Ceph has been instructed to set a memory target of 3G until we increase RAM to 128G per node. Available memory tends to hover around 14G. I do see a tiny bit (KB) of swap utilization per ceph-osd process, but there's no reason for it, so unsure what that's about: root@ceph02:~# cat /proc/14363/status |egrep 'Name|VmSwap' *Name*: ceph-osd *VmSwap*: 464 kB We're seeing repeated inconsistent PG warnings, generally on the order of 3-10 per week. pg 2.b9 is active+clean+inconsistent, acting [25,117,128,95,151,15] PG query on that PG: INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "active+clean+inconsistent", "epoch": 20278, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "acting_recovery_backfill": [ "15(5)", "25(0)", "95(3)", "117(1)", "128(2)", "151(4)" ], "info": { "pgid": "2.b9s0", "last_update": "20278'445510", "last_complete": "20278'445510", "log_tail": "20278'438137", "last_user_version": 445510, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 573, "epoch_pool_created": 100, "last_epoch_started": 14679, "last_interval_started": 14678, "last_epoch_clean": 14716, "last_interval_clean": 14678, "last_epoch_split": 573, "last_epoch_marked_full": 0, "same_up_since": 14678, "same_interval_since": 14678, "same_primary_since": 14396, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "20278'445510", "reported_seq": "896803", "reported_epoch": "20278", "state": "active+clean+inconsistent", "last_fresh": "2020-09-08T18:06:45.463880+0000", "last_change": "2020-09-08T16:57:22.430293+0000", "last_active": "2020-09-08T18:06:45.463880+0000", "last_peered": "2020-09-08T18:06:45.463880+0000", "last_clean": "2020-09-08T18:06:45.463880+0000", "last_became_active": "2020-08-06T19:35:02.634999+0000", "last_became_peered": "2020-08-06T19:35:02.634999+0000", "last_unstale": "2020-09-08T18:06:45.463880+0000", "last_undegraded": "2020-09-08T18:06:45.463880+0000", "last_fullsized": "2020-09-08T18:06:45.463880+0000", "mapping_epoch": 14678, "log_start": "20278'438137", "ondisk_log_start": "20278'438137", "created": 573, "last_epoch_clean": 14716, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "log_size": 7373, "ondisk_log_size": 7373, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 322985947136, "num_objects": 78724, "num_object_clones": 0, "num_object_copies": 472344, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 78724, "num_whiteouts": 0, "num_read": 430713, "num_read_kb": 121695928, "num_write": 445501, "num_write_kb": 405283436, "num_scrub_errors": 1, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 1, "num_objects_recovered": 21, "num_bytes_recovered": 88080384, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 25, "acting_primary": 25, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 14679, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "15(5)", "pgid": "2.b9s5", "last_update": "20278'445510", "last_complete": "18934'278187", "log_tail": "14173'104284", "last_user_version": 111692, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 573, "epoch_pool_created": 100, "last_epoch_started": 14679, "last_interval_started": 14678, "last_epoch_clean": 14716, "last_interval_clean": 14678, "last_epoch_split": 573, "last_epoch_marked_full": 0, "same_up_since": 14678, "same_interval_since": 14678, "same_primary_since": 14396, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "14674'111692", "reported_seq": "127502", "reported_epoch": "14674", "state": "active+undersized+degraded", "last_fresh": "2020-08-06T19:34:47.288930+0000", "last_change": "2020-08-06T19:31:58.919146+0000", "last_active": "2020-08-06T19:34:47.288930+0000", "last_peered": "2020-08-06T19:34:47.288930+0000", "last_clean": "2020-08-06T19:31:36.499168+0000", "last_became_active": "2020-08-06T19:31:58.919146+0000", "last_became_peered": "2020-08-06T19:31:58.919146+0000", "last_unstale": "2020-08-06T19:34:47.288930+0000", "last_undegraded": "2020-08-06T19:31:58.906847+0000", "last_fullsized": "2020-08-06T19:31:58.906728+0000", "mapping_epoch": 14678, "log_start": "14173'104284", "ondisk_log_start": "14173'104284", "created": 573, "last_epoch_clean": 14624, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "14341'106257", "last_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "last_deep_scrub": "14005'91363", "last_deep_scrub_stamp": "2020-08-04T13:36:30.857877+0000", "last_clean_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "log_size": 7408, "ondisk_log_size": 7408, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 83721076736, "num_objects": 19967, "num_object_clones": 0, "num_object_copies": 119802, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 19967, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 19967, "num_whiteouts": 0, "num_read": 938, "num_read_kb": 288244, "num_write": 111692, "num_write_kb": 84295064, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 17, "num_bytes_recovered": 71303168, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "avail_no_missing": [ "25(0)", "15(5)", "95(3)", "117(1)", "128(2)" ], "object_location_counts": [ { "shards": "15(5),25(0),95(3),117(1),128(2)", "objects": 19967 } ], "blocked_by": [], "up_primary": 25, "acting_primary": 25, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 14679, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "95(3)", "pgid": "2.b9s3", "last_update": "20278'445510", "last_complete": "18934'278187", "log_tail": "14173'104284", "last_user_version": 111692, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 573, "epoch_pool_created": 100, "last_epoch_started": 14679, "last_interval_started": 14678, "last_epoch_clean": 14716, "last_interval_clean": 14678, "last_epoch_split": 573, "last_epoch_marked_full": 0, "same_up_since": 14678, "same_interval_since": 14678, "same_primary_since": 14396, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "14674'111692", "reported_seq": "127502", "reported_epoch": "14674", "state": "active+undersized+degraded", "last_fresh": "2020-08-06T19:34:47.288930+0000", "last_change": "2020-08-06T19:31:58.919146+0000", "last_active": "2020-08-06T19:34:47.288930+0000", "last_peered": "2020-08-06T19:34:47.288930+0000", "last_clean": "2020-08-06T19:31:36.499168+0000", "last_became_active": "2020-08-06T19:31:58.919146+0000", "last_became_peered": "2020-08-06T19:31:58.919146+0000", "last_unstale": "2020-08-06T19:34:47.288930+0000", "last_undegraded": "2020-08-06T19:31:58.906847+0000", "last_fullsized": "2020-08-06T19:31:58.906728+0000", "mapping_epoch": 14678, "log_start": "14173'104284", "ondisk_log_start": "14173'104284", "created": 573, "last_epoch_clean": 14624, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "14341'106257", "last_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "last_deep_scrub": "14005'91363", "last_deep_scrub_stamp": "2020-08-04T13:36:30.857877+0000", "last_clean_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "log_size": 7408, "ondisk_log_size": 7408, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 83721076736, "num_objects": 19967, "num_object_clones": 0, "num_object_copies": 119802, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 19967, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 19967, "num_whiteouts": 0, "num_read": 938, "num_read_kb": 288244, "num_write": 111692, "num_write_kb": 84295064, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 17, "num_bytes_recovered": 71303168, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "avail_no_missing": [ "25(0)", "15(5)", "95(3)", "117(1)", "128(2)" ], "object_location_counts": [ { "shards": "15(5),25(0),95(3),117(1),128(2)", "objects": 19967 } ], "blocked_by": [], "up_primary": 25, "acting_primary": 25, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 14679, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "117(1)", "pgid": "2.b9s1", "last_update": "20278'445510", "last_complete": "18934'278187", "log_tail": "14173'104284", "last_user_version": 111692, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 573, "epoch_pool_created": 100, "last_epoch_started": 14679, "last_interval_started": 14678, "last_epoch_clean": 14716, "last_interval_clean": 14678, "last_epoch_split": 573, "last_epoch_marked_full": 0, "same_up_since": 14678, "same_interval_since": 14678, "same_primary_since": 14396, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "14674'111692", "reported_seq": "127502", "reported_epoch": "14674", "state": "active+undersized+degraded", "last_fresh": "2020-08-06T19:34:47.288930+0000", "last_change": "2020-08-06T19:31:58.919146+0000", "last_active": "2020-08-06T19:34:47.288930+0000", "last_peered": "2020-08-06T19:34:47.288930+0000", "last_clean": "2020-08-06T19:31:36.499168+0000", "last_became_active": "2020-08-06T19:31:58.919146+0000", "last_became_peered": "2020-08-06T19:31:58.919146+0000", "last_unstale": "2020-08-06T19:34:47.288930+0000", "last_undegraded": "2020-08-06T19:31:58.906847+0000", "last_fullsized": "2020-08-06T19:31:58.906728+0000", "mapping_epoch": 14678, "log_start": "14173'104284", "ondisk_log_start": "14173'104284", "created": 573, "last_epoch_clean": 14624, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "14341'106257", "last_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "last_deep_scrub": "14005'91363", "last_deep_scrub_stamp": "2020-08-04T13:36:30.857877+0000", "last_clean_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "log_size": 7408, "ondisk_log_size": 7408, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 83721076736, "num_objects": 19967, "num_object_clones": 0, "num_object_copies": 119802, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 19967, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 19967, "num_whiteouts": 0, "num_read": 938, "num_read_kb": 288244, "num_write": 111692, "num_write_kb": 84295064, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 17, "num_bytes_recovered": 71303168, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "avail_no_missing": [ "25(0)", "15(5)", "95(3)", "117(1)", "128(2)" ], "object_location_counts": [ { "shards": "15(5),25(0),95(3),117(1),128(2)", "objects": 19967 } ], "blocked_by": [], "up_primary": 25, "acting_primary": 25, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 14679, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "128(2)", "pgid": "2.b9s2", "last_update": "20278'445510", "last_complete": "18934'278187", "log_tail": "14173'104284", "last_user_version": 111692, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 573, "epoch_pool_created": 100, "last_epoch_started": 14679, "last_interval_started": 14678, "last_epoch_clean": 14716, "last_interval_clean": 14678, "last_epoch_split": 573, "last_epoch_marked_full": 0, "same_up_since": 14678, "same_interval_since": 14678, "same_primary_since": 14396, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "14674'111692", "reported_seq": "127502", "reported_epoch": "14674", "state": "active+undersized+degraded", "last_fresh": "2020-08-06T19:34:47.288930+0000", "last_change": "2020-08-06T19:31:58.919146+0000", "last_active": "2020-08-06T19:34:47.288930+0000", "last_peered": "2020-08-06T19:34:47.288930+0000", "last_clean": "2020-08-06T19:31:36.499168+0000", "last_became_active": "2020-08-06T19:31:58.919146+0000", "last_became_peered": "2020-08-06T19:31:58.919146+0000", "last_unstale": "2020-08-06T19:34:47.288930+0000", "last_undegraded": "2020-08-06T19:31:58.906847+0000", "last_fullsized": "2020-08-06T19:31:58.906728+0000", "mapping_epoch": 14678, "log_start": "14173'104284", "ondisk_log_start": "14173'104284", "created": 573, "last_epoch_clean": 14624, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "14341'106257", "last_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "last_deep_scrub": "14005'91363", "last_deep_scrub_stamp": "2020-08-04T13:36:30.857877+0000", "last_clean_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "log_size": 7408, "ondisk_log_size": 7408, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 83721076736, "num_objects": 19967, "num_object_clones": 0, "num_object_copies": 119802, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 19967, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 19967, "num_whiteouts": 0, "num_read": 938, "num_read_kb": 288244, "num_write": 111692, "num_write_kb": 84295064, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 17, "num_bytes_recovered": 71303168, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "avail_no_missing": [ "25(0)", "15(5)", "95(3)", "117(1)", "128(2)" ], "object_location_counts": [ { "shards": "15(5),25(0),95(3),117(1),128(2)", "objects": 19967 } ], "blocked_by": [], "up_primary": 25, "acting_primary": 25, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 14679, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "151(4)", "pgid": "2.b9s4", "last_update": "20278'445510", "last_complete": "14671'111684", "log_tail": "14173'104284", "last_user_version": 111684, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 573, "epoch_pool_created": 100, "last_epoch_started": 14679, "last_interval_started": 14678, "last_epoch_clean": 14716, "last_interval_clean": 14678, "last_epoch_split": 573, "last_epoch_marked_full": 0, "same_up_since": 14678, "same_interval_since": 14678, "same_primary_since": 14396, "last_scrub": "20278'444009", "last_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_deep_scrub": "20278'444009", "last_deep_scrub_stamp": "2020-09-08T16:57:22.430246+0000", "last_clean_scrub_stamp": "2020-09-07T06:34:26.320796+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "14671'111684", "reported_seq": "127482", "reported_epoch": "14671", "state": "active+clean", "last_fresh": "2020-08-06T19:31:36.499168+0000", "last_change": "2020-08-06T19:28:38.923454+0000", "last_active": "2020-08-06T19:31:36.499168+0000", "last_peered": "2020-08-06T19:31:36.499168+0000", "last_clean": "2020-08-06T19:31:36.499168+0000", "last_became_active": "2020-08-06T19:28:15.372420+0000", "last_became_peered": "2020-08-06T19:28:15.372420+0000", "last_unstale": "2020-08-06T19:31:36.499168+0000", "last_undegraded": "2020-08-06T19:31:36.499168+0000", "last_fullsized": "2020-08-06T19:31:36.499168+0000", "mapping_epoch": 14678, "log_start": "14173'104284", "ondisk_log_start": "14173'104284", "created": 573, "last_epoch_clean": 14624, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "14341'106257", "last_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "last_deep_scrub": "14005'91363", "last_deep_scrub_stamp": "2020-08-04T13:36:30.857877+0000", "last_clean_scrub_stamp": "2020-08-06T00:08:25.447555+0000", "log_size": 7400, "ondisk_log_size": 7400, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 83704299520, "num_objects": 19963, "num_object_clones": 0, "num_object_copies": 119778, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 19963, "num_whiteouts": 0, "num_read": 938, "num_read_kb": 288244, "num_write": 111684, "num_write_kb": 84278680, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 17, "num_bytes_recovered": 71303168, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 25, 117, 128, 95, 151, 15 ], "acting": [ 25, 117, 128, 95, 151, 15 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 25, "acting_primary": 25, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 14679, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2020-08-06T19:35:02.502760+0000", "might_have_unfound": [ { "osd": "15(5)", "status": "already probed" }, { "osd": "95(3)", "status": "already probed" }, { "osd": "117(1)", "status": "already probed" }, { "osd": "128(2)", "status": "already probed" }, { "osd": "151(4)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } } }, { "name": "Started", "enter_time": "2020-08-06T19:35:01.477315+0000" }, { "scrubber.epoch_start": "14678", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } ], "agent_state": {} } Every time we look at them, we see the same checksum (0x6706be76): debug 2020-08-13T18:39:01.731+0000 7fbc037a7700 -1 bluestore(/var/lib/ceph/osd/ceph-25) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0x61f2021c, device location [0x12b403c0000~1000], logical extent 0x0~1000, object 2#2:0f1a338f:::rbd_data.3.20d195d612942.0000000001db869b:head# This looks a lot like: https://tracker.ceph.com/issues/22464 That said, we've got the following versions in play (cluster was created with 15.2.3): ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) This is a containerized cephadm installation, in case it's relevant. Distribution is Ubuntu 18.04.04, kernel is the HWE kernel: Linux ceph02 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux A repair operation 'fixes' it. These are occurring across many PGs, on the various different servers, and we see no indication of any hardware related issues. Any ideas what to do next?

5 8

ceph rbox test on passive compressed pool
by Marc Roos 16 Sep '20

16 Sep '20

I have been inserting 10790 exactly the same 64kb text message to a passive compressing enabled pool. I am still counting, but it looks like only half the objects are compressed. mail/b08c3218dbf1545ff430000052412a8e mtime 2020-09-06 16:27:39.000000, size 63580 mail/00f6043775f1545ff430000052412a8e mtime 2020-09-06 16:25:57.000000, size 525 mail/b875f40571f1545ff430000052412a8e mtime 2020-09-06 16:25:53.000000, size 63580 mail/e87c120b19f1545ff430000052412a8e mtime 2020-09-06 16:24:25.000000, size 525 I am not sure if this should be expected from passive, these docs[1] hint that passive 'compress if hinted COMPRESSIBLE'. From that I would conclude that all text messages should be compressed. A previous test with a 64kb gzip attachment seemed to not compress, although I did not look at all object sizes. on 14.2.11 [1] https://documentation.suse.com/ses/5.5/html/ses-all/ceph-pools.html#sec-cep… https://docs.ceph.com/docs/mimic/rados/operations/pools/

3 8

Matlab Assignment Help
by brainycoordinator＠gmail.com 16 Sep '20

16 Sep '20

MATLAB is the easiest software and it is the most productive software. But students are unable to handle this software and look for <a href="https://sourceessay.com/online-assignment-help-assistance/">Online Assignment Help</a>. If you are looking for assignment help then you can hire best assignment writers. These writers are very professional and know all types of software and know how to complete the assignment in a professional way.

1 0

How does the sign-in issue stop to Unlock Yahoo Account?
by mary smith 16 Sep '20

16 Sep '20

On the off chance that you get incapable to sign in to your Yahoo account, at that point you have to apply the right investigating steps to keep away from Unlock Yahoo Account circumstance. All things considered, on the off chance that you entered an inappropriate login certification, at that point your account will be blocked, and afterward, you have to unlock it to utilize it for additional. https://www.customercare-email.com/blog/unlock-my-yahoo-account/

1 0

Can't do creating plant settings in Epson contraption? Call Epson Printer Support.
by mary smith 16 Sep '20

16 Sep '20

The gadget is known for dealing with an assortment of documents. Regardless, the records can break down and henceforth the need may ascend to fix the issue. Considering, you can utilize the help that is given in the specific assistance zones or you can find uphold from the Epson Printer Support. https://www.epsonprintersupportpro.net/

1 0

rbd-nbd multi queue
by Bobby 15 Sep '20

15 Sep '20

Hi, I have come across an old thread (2017) on the topic rbd-nbd performance. Here: https://www.spinics.net/lists/ceph-devel/msg36645.html It says they have tried adding multi-connections support on rbd-nbd with newest nbd driver, so that the nbd driver can create multiple io queues, and each io queue is associated with one socket connection to talk to rbd-nbd for request sending and response receiving. I have to work on the simular use-case. My question is , does the current rbd-ndb tool has the multi-connections support by default so that I can create multiple IO queues in the nbd driver? Thanks Bobby

1 0

Syncing cephfs from Ceph to Ceph
by Simon Sutter 15 Sep '20

15 Sep '20

Hello, Is it possible to somehow sync a ceph from one site to a ceph form another site? I'm just using the cephfs feature and no block devices. Being able to sync cephfs pools between two sites would be great for a hot backup, in case one site fails. Thanks in advance, Simon

4 5

Unable to start mds when creating cephfs volume with erasure encoding data pool
by tri＠postix.net 15 Sep '20

15 Sep '20

Hi all, I'm using ceph Octopus version and deployed it using cephadm. The ceph documentation provides 2 ways for creating a new cephfs volume: 1. via "ceph fs volume create ..." - I can use this and it works fine with the MDS automatically deployed but there is no provision for using EC with the data pool 2. via "ceph fs new ..." - I can create the new fs volume but the MDS wasn't automatically launched. If I launch them manually using "ceph orch daemon add mds ...", the mds got terminated immediated as orphan so the new fs volume isn't usable. I'd very much appreciate any pointer to how to properly create a new cephfs volume with an erasure encoding data pool. Best regards, --THoang

3 2

Re: benchmark Ceph
by rainning 15 Sep '20

15 Sep '20

What is your Ceph version? From the test results you posted, your environment's performance is okay in regard of your setup. But there are definitely many things that can be tuned to get you better number. I normally use top, iostat, pidstat, vmstat, dstat, iperf3, blktrace, netmon, ceph admin socket to monitor system stat. ------------------ Original ------------------ From:  "Tony Liu";<tonyliu0592(a)hotmail.com>; Date:  Sep 15, 2020 To:  "rainning"<tweetypie(a)qq.com>; "ceph-users"<ceph-users(a)ceph.io>; Subject:  [ceph-users] Re: benchmark Ceph Here is the test inside VM. ================ # fio --name=test --ioengine=libaio --numjobs=1 --runtime=30 \                   --direct=1 --size=2G --end_fsync=1 \                   --rw=read --bs=4K --iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=14615: Mon Sep 14 21:50:55 2020    read: IOPS=3209, BW=12.5MiB/s (13.1MB/s)(376MiB/30001msec)     slat (usec): min=3, max=162, avg= 6.91, stdev= 4.74     clat (usec): min=85, max=17366, avg=303.17, stdev=639.42      lat (usec): min=161, max=17373, avg=310.38, stdev=639.93     clat percentiles (usec):      |  1.00th=[  167],  5.00th=[  172], 10.00th=[  176], 20.00th=[  182],      | 30.00th=[  188], 40.00th=[  194], 50.00th=[  204], 60.00th=[  221],      | 70.00th=[  239], 80.00th=[  277], 90.00th=[  359], 95.00th=[  461],      | 99.00th=[ 3130], 99.50th=[ 5735], 99.90th=[ 8094], 99.95th=[11338],      | 99.99th=[14091]    bw (  KiB/s): min= 9688, max=15120, per=99.87%, avg=12820.51, stdev=1001.88, samples=59    iops        : min= 2422, max= 3780, avg=3205.12, stdev=250.47, samples=59   lat (usec)   : 100=0.01%, 250=74.99%, 500=20.76%, 750=2.21%, 1000=0.50%   lat (msec)   : 2=0.39%, 4=0.27%, 10=0.81%, 20=0.06%   cpu          : usr=0.65%, sys=3.06%, ctx=96287, majf=0, minf=13   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=96287,0,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):    READ: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=376MiB (394MB), run=30001-30001msec Disk stats (read/write):   vda: ios=95957/2, merge=0/0, ticks=29225/12, in_queue=6027, util=82.52% ================ ================ # fio --name=test --ioengine=libaio --numjobs=1 --runtime=30 \                   --direct=1 --size=2G --end_fsync=1 \                   --rw=write --bs=4K --iodepth=1 test: (groupid=0, jobs=1): err= 0: pid=14619: Mon Sep 14 21:52:04 2020   write: IOPS=16.3k, BW=63.7MiB/s (66.8MB/s)(1917MiB/30074msec)     slat (usec): min=3, max=182, avg= 5.94, stdev= 1.30     clat (usec): min=11, max=5234, avg=54.08, stdev=18.58      lat (usec): min=35, max=5254, avg=60.26, stdev=18.80     clat percentiles (usec):      |  1.00th=[   36],  5.00th=[   38], 10.00th=[   40], 20.00th=[   46],      | 30.00th=[   48], 40.00th=[   50], 50.00th=[   53], 60.00th=[   56],      | 70.00th=[   59], 80.00th=[   63], 90.00th=[   67], 95.00th=[   71],      | 99.00th=[   85], 99.50th=[  100], 99.90th=[  289], 99.95th=[  355],      | 99.99th=[  412]    bw (  KiB/s): min=59640, max=80982, per=100.00%, avg=65462.25, stdev=7166.81, samples=59    iops        : min=14910, max=20245, avg=16365.54, stdev=1791.69, samples=59   lat (usec)   : 20=0.01%, 50=39.85%, 100=59.65%, 250=0.36%, 500=0.14%   lat (usec)   : 750=0.01%, 1000=0.01%   lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%   cpu          : usr=2.10%, sys=11.63%, ctx=490639, majf=0, minf=12   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=0,490635,0,1 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):   WRITE: bw=63.7MiB/s (66.8MB/s), 63.7MiB/s-63.7MiB/s (66.8MB/s-66.8MB/s), io=1917MiB (2010MB), run=30074-30074msec Disk stats (read/write):   vda: ios=9/490639, merge=0/0, ticks=26/27102, in_queue=184, util=99.36% ================ Both networking and storage workloads are light. Which system stat should I monitor? Thanks! Tony > -----Original Message----- > From: rainning <tweetypie(a)qq.com> > Sent: Monday, September 14, 2020 8:39 PM > To: Tony Liu <tonyliu0592(a)hotmail.com>; ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: benchmark Ceph > > Can you post the fio results with the ioengine using libaio? From what > you posted, it seems to me that the read test hit cache. And the write > performance was not good, the latency was too high (~35.4ms) while the > numjobs and iodepth both were 1. Did you monitor system stat on both > side (VM/Compute Node and Cluster)? > > > > > ------------------&nbsp;Original&nbsp;------------------ > From: &nbsp;"Tony Liu";<tonyliu0592(a)hotmail.com&gt;; > Date: &nbsp;Sep 15, 2020 > To: &nbsp;"ceph-users"<ceph-users(a)ceph.io&gt;; > > Subject: &nbsp;[ceph-users] benchmark Ceph > > > > Hi, > > I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS > HDD on each node, to provide storage to a OpenStack cluster. Both public > and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and > they share the same 60GB partition. > > I run fio with different combinations of operation, block size and io- > depth to collect IOPS, bandwidth and latency. I tried fio on compute > node with ioengine=rbd, also fio within VM (backed by Ceph) with > ioengine=libaio. > > The result doesn't seem good. Here are couple examples. > ==================================== > fio --name=test --ioengine=rbd --clientname=admin \ &nbsp;&nbsp;&nbsp; - > -pool=benchmark --rbdname=test --numjobs=1 \ &nbsp;&nbsp;&nbsp; -- > runtime=30 --direct=1 --size=2G \ &nbsp;&nbsp;&nbsp; --rw=read --bs=4k - > -iodepth=1 > > test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- > 4096B, ioengine=rbd, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020 > &nbsp;&nbsp; read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec) > &nbsp;&nbsp;&nbsp; slat (nsec): min=1550, max=57662, avg=3312.74, > stdev=2981.42 &nbsp;&nbsp;&nbsp; clat (usec): min=77, max=4799, > avg=127.39, stdev=39.88 &nbsp;&nbsp;&nbsp;&nbsp; lat (usec): min=78, > max=4812, avg=130.70, stdev=40.67 &nbsp;&nbsp;&nbsp; clat percentiles > (usec): > &nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; 1.00th=[&nbsp;&nbsp; 82],&nbsp; > 5.00th=[&nbsp;&nbsp; 86], 10.00th=[&nbsp;&nbsp; 95], > 20.00th=[&nbsp;&nbsp; 98], &nbsp;&nbsp;&nbsp;&nbsp; | 30.00th=[&nbsp; > 100], 40.00th=[&nbsp; 104], 50.00th=[&nbsp; 116], 60.00th=[&nbsp; 129], > &nbsp;&nbsp;&nbsp;&nbsp; | 70.00th=[&nbsp; 141], 80.00th=[&nbsp; 157], > 90.00th=[&nbsp; 182], 95.00th=[&nbsp; 198], &nbsp;&nbsp;&nbsp;&nbsp; | > 99.00th=[&nbsp; 233], 99.50th=[&nbsp; 245], 99.90th=[&nbsp; 359], > 99.95th=[&nbsp; 515], &nbsp;&nbsp;&nbsp;&nbsp; | 99.99th=[&nbsp; 709] > &nbsp;&nbsp; bw (&nbsp; KiB/s): min=27160, max=40696, per=100.00%, > avg=30474.29, stdev=2826.23, samples=59 &nbsp;&nbsp; > iops&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : min= 6790, max=10174, > avg=7618.56, stdev=706.56, samples=59 &nbsp; lat (usec)&nbsp;&nbsp; : > 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01% &nbsp; lat > (msec)&nbsp;&nbsp; : 2=0.01%, 10=0.01% &nbsp; > cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : usr=3.55%, > sys=3.80%, ctx=228358, majf=0, minf=29 &nbsp; IO > depths&nbsp;&nbsp;&nbsp; : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; submit&nbsp;&nbsp;&nbsp; : > 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% > &nbsp;&nbsp;&nbsp;&nbsp; complete&nbsp; : 0=0.0%, 4=100.0%, 8=0.0%, > 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; issued > rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0 > &nbsp;&nbsp;&nbsp;&nbsp; latency&nbsp;&nbsp; : target=0, window=0, > percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > &nbsp;&nbsp; READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s > (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec > > Disk stats (read/write): > &nbsp;&nbsp;&nbsp; dm-0: ios=290/3, merge=0/0, ticks=2427/19, > in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, > aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95% &nbsp; sda: > ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% > ==================================== > ==================================== > fio --name=test --ioengine=rbd --clientname=admin \ &nbsp;&nbsp;&nbsp; - > -pool=benchmark --rbdname=test --numjobs=1 \ &nbsp;&nbsp;&nbsp; -- > runtime=30 --direct=1 --size=2G \ &nbsp;&nbsp;&nbsp; --rw=write --bs=4k > --iodepth=1 > > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B- > 4096B, ioengine=rbd, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020 > &nbsp; write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec) > &nbsp;&nbsp;&nbsp; slat (nsec): min=2240, max=45925, avg=6526.95, > stdev=3486.19 &nbsp;&nbsp;&nbsp; clat (usec): min=399, max=35411, > avg=615.88, stdev=231.41 &nbsp;&nbsp;&nbsp;&nbsp; lat (usec): min=402, > max=35421, avg=622.40, stdev=232.08 &nbsp;&nbsp;&nbsp; clat percentiles > (usec): > &nbsp;&nbsp;&nbsp;&nbsp; |&nbsp; 1.00th=[&nbsp; 420],&nbsp; > 5.00th=[&nbsp; 449], 10.00th=[&nbsp; 469], 20.00th=[&nbsp; 498], > &nbsp;&nbsp;&nbsp;&nbsp; | 30.00th=[&nbsp; 529], 40.00th=[&nbsp; 562], > 50.00th=[&nbsp; 611], 60.00th=[&nbsp; 652], &nbsp;&nbsp;&nbsp;&nbsp; | > 70.00th=[&nbsp; 685], 80.00th=[&nbsp; 709], 90.00th=[&nbsp; 766], > 95.00th=[&nbsp; 799], &nbsp;&nbsp;&nbsp;&nbsp; | 99.00th=[&nbsp; 881], > 99.50th=[&nbsp; 955], 99.90th=[ 2671], 99.95th=[ 3097], > &nbsp;&nbsp;&nbsp;&nbsp; | 99.99th=[ 3785] &nbsp;&nbsp; bw (&nbsp; > KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, > samples=60 &nbsp;&nbsp; iops&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : > min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60 &nbsp; lat > (usec)&nbsp;&nbsp; : 500=20.82%, 750=67.23%, 1000=11.55% &nbsp; lat > (msec)&nbsp;&nbsp; : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01% > &nbsp; cpu&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : > usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18 &nbsp; IO > depths&nbsp;&nbsp;&nbsp; : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, > 32=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; submit&nbsp;&nbsp;&nbsp; : > 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% > &nbsp;&nbsp;&nbsp;&nbsp; complete&nbsp; : 0=0.0%, 4=100.0%, 8=0.0%, > 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% &nbsp;&nbsp;&nbsp;&nbsp; issued > rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0 > &nbsp;&nbsp;&nbsp;&nbsp; latency&nbsp;&nbsp; : target=0, window=0, > percentile=100.00%, depth=1 > > Run status group 0 (all jobs): > &nbsp; WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s- > 6571kB/s), io=188MiB (197MB), run=30003-30003msec > > Disk stats (read/write): > &nbsp;&nbsp;&nbsp; dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, > util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, > aggrin_queue=404, aggrutil=0.13% &nbsp; sda: ios=33/3, merge=0/0, > ticks=390/27, in_queue=404, util=0.13% > ==================================== > > Does that make sense? How do you benchmark your Ceph cluster? > Appreciate if you could share your experiences here. > > Thanks! > Tony > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Nautilus: rbd image stuck unaccessible after VM restart
by islepnev＠gmail.com 15 Sep '20

15 Sep '20

Hello, I’m running kvm virtualization with rbd storage, some images on rbd pool become efficiently unusable after VM restart. All I/O to problematic rbd image blocks infinitely. Checked that it is not a permission or locking problem. The bug was silent until we performed a planned restart of few VMs and some of VMs failed to start (kvm process timed out). It could be related to recent upgrades luminous to nautilus or proxmox 5 to 6. Ceph backend is clean, no observable problems, all mons/mgrs/osds up and running. Network is ok. Nothing in logs relevant to the problem. ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus (stable) kernel 5.3.13-2-pve #1 SMP PVE 5.3.13-2 (Fri, 24 Jan 2020 09:49:36 +0100) x86_64 GNU/Linux HEALTH_OK No locks: # rbd status rbd-technet/vm-402-disk-0 Watchers: none # rbd status rbd-technet/vm-402-disk-1 Watchers: none Normal image vs problematic: # rbd object-map check rbd-technet/vm-402-disk-0 Object Map Check: 100% complete…done. # rbd object-map check rbd-technet/vm-402-disk-1 ^C disk-0 is good while disk-1 is effectively lost. Command hangs for many minutes with no visible activity, interrupted. rbd export runs without problems, however some data is lost after being imported back (ext4 errors). rbd deep copy worked for me. Copy looks good, no errors. # rbd info rbd-technet/vm-402-disk-1 rbd image 'vm-402-disk-1': size 16 GiB in 4096 objects order 22 (4 MiB objects) snapshot_count: 0 id: c600d06b8b4567 block_name_prefix: rbd_data.c600d06b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Fri Jan 31 17:50:50 2020 access_timestamp: Sat Mar 7 00:30:53 2020 modify_timestamp: Sat Mar 7 00:33:35 2020 journal: c600d06b8b4567 mirroring state: disabled What can be done to debug this problem? Thanks, Ilia.

4 3

Re: benchmark Ceph
by rainning 15 Sep '20

15 Sep '20

Can you post the fio results with the ioengine using libaio? From what you posted, it seems to me that the read test hit cache. And the write performance was not good, the latency was too high (~35.4ms) while the numjobs and iodepth both were 1. Did you monitor system stat on both side (VM/Compute Node and Cluster)? ------------------ Original ------------------ From:  "Tony Liu";<tonyliu0592(a)hotmail.com>; Date:  Sep 15, 2020 To:  "ceph-users"<ceph-users(a)ceph.io>; Subject:  [ceph-users] benchmark Ceph Hi, I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS HDD on each node, to provide storage to a OpenStack cluster. Both public and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and they share the same 60GB partition. I run fio with different combinations of operation, block size and io-depth to collect IOPS, bandwidth and latency. I tried fio on compute node with ioengine=rbd, also fio within VM (backed by Ceph) with ioengine=libaio. The result doesn't seem good. Here are couple examples. ==================================== fio --name=test --ioengine=rbd --clientname=admin \     --pool=benchmark --rbdname=test --numjobs=1 \     --runtime=30 --direct=1 --size=2G \     --rw=read --bs=4k --iodepth=1 test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020    read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec)     slat (nsec): min=1550, max=57662, avg=3312.74, stdev=2981.42     clat (usec): min=77, max=4799, avg=127.39, stdev=39.88      lat (usec): min=78, max=4812, avg=130.70, stdev=40.67     clat percentiles (usec):      |  1.00th=[   82],  5.00th=[   86], 10.00th=[   95], 20.00th=[   98],      | 30.00th=[  100], 40.00th=[  104], 50.00th=[  116], 60.00th=[  129],      | 70.00th=[  141], 80.00th=[  157], 90.00th=[  182], 95.00th=[  198],      | 99.00th=[  233], 99.50th=[  245], 99.90th=[  359], 99.95th=[  515],      | 99.99th=[  709]    bw (  KiB/s): min=27160, max=40696, per=100.00%, avg=30474.29, stdev=2826.23, samples=59    iops        : min= 6790, max=10174, avg=7618.56, stdev=706.56, samples=59   lat (usec)   : 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01%   lat (msec)   : 2=0.01%, 10=0.01%   cpu          : usr=3.55%, sys=3.80%, ctx=228358, majf=0, minf=29   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):    READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec Disk stats (read/write):     dm-0: ios=290/3, merge=0/0, ticks=2427/19, in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95%   sda: ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% ==================================== ==================================== fio --name=test --ioengine=rbd --clientname=admin \     --pool=benchmark --rbdname=test --numjobs=1 \     --runtime=30 --direct=1 --size=2G \     --rw=write --bs=4k --iodepth=1 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020   write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec)     slat (nsec): min=2240, max=45925, avg=6526.95, stdev=3486.19     clat (usec): min=399, max=35411, avg=615.88, stdev=231.41      lat (usec): min=402, max=35421, avg=622.40, stdev=232.08     clat percentiles (usec):      |  1.00th=[  420],  5.00th=[  449], 10.00th=[  469], 20.00th=[  498],      | 30.00th=[  529], 40.00th=[  562], 50.00th=[  611], 60.00th=[  652],      | 70.00th=[  685], 80.00th=[  709], 90.00th=[  766], 95.00th=[  799],      | 99.00th=[  881], 99.50th=[  955], 99.90th=[ 2671], 99.95th=[ 3097],      | 99.99th=[ 3785]    bw (  KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, samples=60    iops        : min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60   lat (usec)   : 500=20.82%, 750=67.23%, 1000=11.55%   lat (msec)   : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01%   cpu          : usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%      issued rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0      latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):   WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s-6571kB/s), io=188MiB (197MB), run=30003-30003msec Disk stats (read/write):     dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, aggrin_queue=404, aggrutil=0.13%   sda: ios=33/3, merge=0/0, ticks=390/27, in_queue=404, util=0.13% ==================================== Does that make sense? How do you benchmark your Ceph cluster? Appreciate if you could share your experiences here. Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

benchmark Ceph
by Tony Liu 15 Sep '20

15 Sep '20

Hi, I have a 3-OSD-node Ceph cluster with 1 480GB SSD and 8 x 2TB 12Gpbs SAS HDD on each node, to provide storage to a OpenStack cluster. Both public and cluster networks are 2x10G. WAL and DB of each OSD is on SSD and they share the same 60GB partition. I run fio with different combinations of operation, block size and io-depth to collect IOPS, bandwidth and latency. I tried fio on compute node with ioengine=rbd, also fio within VM (backed by Ceph) with ioengine=libaio. The result doesn't seem good. Here are couple examples. ==================================== fio --name=test --ioengine=rbd --clientname=admin \ --pool=benchmark --rbdname=test --numjobs=1 \ --runtime=30 --direct=1 --size=2G \ --rw=read --bs=4k --iodepth=1 test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=0): [f(1)][100.0%][r=27.6MiB/s,w=0KiB/s][r=7075,w=0 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56310: Mon Sep 14 19:01:24 2020 read: IOPS=7610, BW=29.7MiB/s (31.2MB/s)(892MiB/30001msec) slat (nsec): min=1550, max=57662, avg=3312.74, stdev=2981.42 clat (usec): min=77, max=4799, avg=127.39, stdev=39.88 lat (usec): min=78, max=4812, avg=130.70, stdev=40.67 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 86], 10.00th=[ 95], 20.00th=[ 98], | 30.00th=[ 100], 40.00th=[ 104], 50.00th=[ 116], 60.00th=[ 129], | 70.00th=[ 141], 80.00th=[ 157], 90.00th=[ 182], 95.00th=[ 198], | 99.00th=[ 233], 99.50th=[ 245], 99.90th=[ 359], 99.95th=[ 515], | 99.99th=[ 709] bw ( KiB/s): min=27160, max=40696, per=100.00%, avg=30474.29, stdev=2826.23, samples=59 iops : min= 6790, max=10174, avg=7618.56, stdev=706.56, samples=59 lat (usec) : 100=28.89%, 250=70.72%, 500=0.34%, 750=0.05%, 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=3.55%, sys=3.80%, ctx=228358, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=228333,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=29.7MiB/s (31.2MB/s), 29.7MiB/s-29.7MiB/s (31.2MB/s-31.2MB/s), io=892MiB (935MB), run=30001-30001msec Disk stats (read/write): dm-0: ios=290/3, merge=0/0, ticks=2427/19, in_queue=2446, util=0.95%, aggrios=290/4, aggrmerge=0/0, aggrticks=2427/39, aggrin_queue=2332, aggrutil=0.95% sda: ios=290/4, merge=0/0, ticks=2427/39, in_queue=2332, util=0.95% ==================================== ==================================== fio --name=test --ioengine=rbd --clientname=admin \ --pool=benchmark --rbdname=test --numjobs=1 \ --runtime=30 --direct=1 --size=2G \ --rw=write --bs=4k --iodepth=1 test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-3.7 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=6352KiB/s][r=0,w=1588 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=56544: Mon Sep 14 19:03:36 2020 write: IOPS=1604, BW=6417KiB/s (6571kB/s)(188MiB/30003msec) slat (nsec): min=2240, max=45925, avg=6526.95, stdev=3486.19 clat (usec): min=399, max=35411, avg=615.88, stdev=231.41 lat (usec): min=402, max=35421, avg=622.40, stdev=232.08 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 449], 10.00th=[ 469], 20.00th=[ 498], | 30.00th=[ 529], 40.00th=[ 562], 50.00th=[ 611], 60.00th=[ 652], | 70.00th=[ 685], 80.00th=[ 709], 90.00th=[ 766], 95.00th=[ 799], | 99.00th=[ 881], 99.50th=[ 955], 99.90th=[ 2671], 99.95th=[ 3097], | 99.99th=[ 3785] bw ( KiB/s): min= 5944, max= 6792, per=100.00%, avg=6415.95, stdev=178.72, samples=60 iops : min= 1486, max= 1698, avg=1603.93, stdev=44.67, samples=60 lat (usec) : 500=20.82%, 750=67.23%, 1000=11.55% lat (msec) : 2=0.25%, 4=0.14%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=1.22%, sys=1.25%, ctx=48143, majf=0, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,48129,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=6417KiB/s (6571kB/s), 6417KiB/s-6417KiB/s (6571kB/s-6571kB/s), io=188MiB (197MB), run=30003-30003msec Disk stats (read/write): dm-0: ios=31/2, merge=0/0, ticks=342/14, in_queue=356, util=0.12%, aggrios=33/3, aggrmerge=0/0, aggrticks=390/27, aggrin_queue=404, aggrutil=0.13% sda: ios=33/3, merge=0/0, ticks=390/27, in_queue=404, util=0.13% ==================================== Does that make sense? How do you benchmark your Ceph cluster? Appreciate if you could share your experiences here. Thanks! Tony

1 0

Ceph RBD iSCSI compatibility
by Salsa 14 Sep '20

14 Sep '20

I just came across a Suse documentation stating that RBD features are not iSCSI compatible. Since I had 2 cases of image corruption in this scenario in 10 days I'm wondering if my setup is to blame. So question is if it is possible to provide disks to a Windows Server 2019 via iSCSI while using rbd-mirror to backup data to a second cluster? I created all images with all features enabled. Is that compatible? -- Salsa

6 12

Choosing suitable SSD for Ceph cluster
by Hermann Himmelbauer 14 Sep '20

14 Sep '20

Hi, I am running a nice ceph (proxmox 4 / debian-8 / ceph 0.94.3) cluster on 3 nodes (supermicro X8DTT-HIBQF), 2 OSD each (2TB SATA harddisks), interconnected via Infiniband 40. Problem is that the ceph performance is quite bad (approx. 30MiB/s reading, 3-4 MiB/s writing ), so I thought about plugging into each node a PCIe to NVMe/M.2 adapter and install SSD harddisks. The idea is to have a faster ceph storage and also some storage extension. The question is now which SSDs I should use. If I understand it right, not every SSD is suitable for ceph, as is denoted at the links below: https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-i… or here: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark In the first link, the Samsung SSD 950 PRO 512GB NVMe is listed as a fast SSD for ceph. As the 950 is not available anymore, I ordered a Samsung 970 1TB for testing, unfortunately, the "EVO" instead of PRO. Before equipping all nodes with these SSDs, I did some tests with "fio" as recommended, e.g. like this: fio --filename=/dev/DEVICE --direct=1 --sync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test The results are as the following: ----------------------- 1) Samsung 970 EVO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=26706MB, bw=445MiB/s, iops=113945, runt= 60001msec write: io=252576KB, bw=4.1MiB/s, iops=1052, runt= 60001msec Jobs: 4: read : io=21805MB, bw=432.7MiB/s, iops=93034, runt= 60001msec write: io=422204KB, bw=6.8MiB/s, iops=1759, runt= 60002msec Jobs: 10: read : io=26921MB, bw=448MiB/s, iops=114859, runt= 60001msec write: io=435644KB, bw=7MiB/s, iops=1815, runt= 60004msec ----------------------- So the read speed is impressive, but the write speed is really bad. Therefore I ordered the Samsung 970 PRO (1TB) as it has faster NAND chips (MLC instead of TLC). The results are, however even worse for writing: ----------------------- Samsung 970 PRO NVMe M.2 mit PCIe Adapter Jobs: 1: read : io=15570MB, bw=259.4MiB/s, iops=66430, runt= 60001msec write: io=199436KB, bw=3.2MiB/s, iops=830, runt= 60001msec Jobs: 4: read : io=48982MB, bw=816.3MiB/s, iops=208986, runt= 60001msec write: io=327800KB, bw=5.3MiB/s, iops=1365, runt= 60002msec Jobs: 10: read : io=91753MB, bw=1529.3MiB/s, iops=391474, runt= 60001msec write: io=343368KB, bw=5.6MiB/s, iops=1430, runt= 60005msec ----------------------- I did some research and found out, that the "--sync" flag sets the flag "O_DSYNC" which seems to disable the SSD cache which leads to these horrid write speeds. It seems that this relates to the fact that the write cache is only not disabled for SSDs which implement some kind of battery buffer that guarantees a data flush to the flash in case of a powerloss. However, It seems impossible to find out which SSDs do have this powerloss protection, moreover, these enterprise SSDs are crazy expensive compared to the SSDs above - moreover it's unclear if powerloss protection is even available in the NVMe form factor. So building a 1 or 2 TB cluster seems not really affordable/viable. So, can please anyone give me hints what to do? Is it possible to ensure that the write cache is not disabled in some way (my server is situated in a data center, so there will probably never be loss of power). Or is the link above already outdated as newer ceph releases somehow deal with this problem? Or maybe a later Debian release (10) will handle the O_DSYNC flag differently? Perhaps I should simply invest in faster (and bigger) harddisks and forget the SSD-cluster idea? Thank you in advance for any help, Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

12 18

ceph-container: docker restart, mon's unable to join
by Stefan Kooman 14 Sep '20

14 Sep '20

Hi, In an attempt to get a (test) Mimic cluster running on Ubuntu 20.04 we are using docker with ceph-container images (ceph/daemon:latest-mimic). Deploying monitors and mgrs works fine. If however a monitor container gets stopped and started (i.e. docker restart) two out of three (with exception of mon initial member) mons won't join the cluster anymore and keep logging the following: /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster... If docker is stopped, the mon directory "/var/lib/ceph/mon/$mon-name" removed and docker started again the mon is able to join the cluster. This directory is a persistent volume with correct permissions (167.167). No etcd cluster is in use here. We manually copied the /etc/ceph and /var/lib/ceph directories to the docker hosts. Any hints on how to make a mon container survive a reboot is welcome. Gr. Stefan P.s And yes, we know about Rook, kubernetes, etc. but that's not what want to use now.

1 0

ceph-osd performance on ram disk
by George Shuklin 14 Sep '20

14 Sep '20

I'm creating a benchmark suite for Сeph. During benchmarking of benchmark, I've checked how fast ceph-osd works. I decided to skip all 'SSD mess' and use brd (block ram disk, modprobe brd) as underlying storage. Brd itself can yield up to 2.7Mpps in fio. In single thread mode (iodepth=1) it can yield up to 750k IOPS. LVM over brd gives about 600kIOPS in single-threaded mode with iodepth=1 (16us latency). But, as soon as I put ceph-osd (bluestore) on it, I see something very odd. No matter how much parallel load I push onto this OSD, it never gives more than 30 kIOPS, and I can't understand where bottleneck is. CPU utilization: ~300%. There are 8 cores on my setup, so, CPU is not a bottleneck. Network: I've moved benchmark on the same host as OSD, so it's a localhost. Even counting network, it's still far away from saturation. 30kIOPS (4k) is about 1Gb/s, but I have 10G links. Anyway, tests are run on localhost, so network is irrelevant (I've checked it, traffic is on localhost). Test itself consumes about 70% CPU of one core, so there are plenty left. Replication: I've killed it (size=1, single osd in the pool). single-threaded latency: 200us, 4.8kIOPS. iopdeth=32: 2ms (15kIOPS). iodepth=16,numjobs=8: 5ms (24k IOPS) I'm running fio with 'rados' ioengine, and it looks like putting more workers doesn't change much, so it's not rados ioengine. As there is plenty CPU and IO left, there is only one possible place for bottleneck: some time-consuming single-threaded code in ceph-osd. Are there any knobs to tweak to see higher performance for ceph-osd? I'm pretty sure it's not any kind of leveling, GC or other 'iops-related' issues (brd has performance of two order of magnitude higher).

3 5

Orchestrator & ceph osd purge
by Robert Sander 14 Sep '20

14 Sep '20

Hi, is it correct that when using the orchestrator to deploy and manage a cluster you should not use "ceph osd purge" any more as the orchestrator then is not able to find the OSD for the "ceph orch osd rm" operation? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Is it possible to assign osd id numbers?
by Shain Miley 14 Sep '20

14 Sep '20

Hello, I have been wondering for quite some time whether or not it is possible to influence the osd.id numbers that are assigned during an install. I have made an attempt to keep our osds in order over the last few years, but it is a losing battle without having some control over the osd assignment. I am currently using ceph-deploy to handle adding nodes to the cluster. Thanks in advance, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org

3 4

Assignment Help
by jack39068＠gmail.com 13 Sep '20

13 Sep '20

We are the best in online assignment help and students stuck in their assignment so don’t worry we are available 24/7. Take best help with our high qualified experts.https://www.fullhomeworkhelp.com/pages/assignment/assignment-help-a…

1 0

Incapable to utilize make include out of Facebook? Call Facebook Customer Service Toll Free Number.
by mary smith 12 Sep '20

12 Sep '20

The make include out of the mail is utilized to compose an email however on the off chance that you can't utilize it, at that point you can get the important assistance from the assistance community by utilizing their FAQs. Notwithstanding that, you can likewise call the Facebook Customer Service Toll Free Number to get the necessary help. https://www.customercare-email.com/facebook-customer-service.html

1 0

Issue in interface causing Epson Error Code 0x97? Get to customer care team for help.
by mary smith 12 Sep '20

12 Sep '20

Interface issues can also lead to Epson Error Code 0x97 as the glitch can cause hindrance with smooth functioning. However, the error can be resolved if you use the help that is provided by customer care. All you have to do is to call the team by dialing the tech support number that is available on the internet.https://www.epsonprintersupportpro.net/epson-error-code-0x97/

1 0

Change crush rule on pool
by jesper＠krogh.cc 12 Sep '20

12 Sep '20

Hi I would like to change the crush rule so data lands on ssd instead of hdd, can this be done on the fly and migration will just happen or do I need to do something to move data? Jesper Sent from myMail for iOS

3 4

The confusing output of ceph df command
by norman kern 12 Sep '20

12 Sep '20

6 8

Re: OSDs and tmpfs
by Shain Miley 11 Sep '20

11 Sep '20

I am going to attempt to answer my own question here and someone can correct me if I am wrong. Looking at a few of the other OSDs that we have replaced over the last year or so it looks like they are mounted using tmpfs as well and that this is just a result of switching from filestore to bluestore and that this is really nothing to worry about. Thanks, Shain On 9/9/20, 11:16 AM, "Shain Miley" <SMiley(a)npr.org> wrote: Hi, I recently added 3 new servers to Ceph cluster. These servers use the H740p mini raid card and I had to install the HWE kernel in Ubuntu 16.04 in order to get the drives recognized. We have a 23 node cluster and normally when we add OSDs they end up mounting like this: /dev/sde1 3.7T 2.0T 1.8T 54% /var/lib/ceph/osd/ceph-15 /dev/sdj1 3.7T 2.0T 1.7T 55% /var/lib/ceph/osd/ceph-20 /dev/sdd1 3.7T 2.1T 1.6T 58% /var/lib/ceph/osd/ceph-14 /dev/sdc1 3.7T 1.8T 1.9T 49% /var/lib/ceph/osd/ceph-13 However I noticed this morning that the 3 new servers have the OSDs mounted like this: tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-246 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-240 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-248 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-237 Is this normal for deployments going forward…or did something go wrong? These are 12TB drives but they are showing up as 47G here instead. We are using ceph version 12.2.13 and I installed this using ceph-deply version 2.0.1. Thanks in advance, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 3

Problem unusable after deleting pool with bilion objects
by Jan Pekař - Imatic 11 Sep '20

11 Sep '20

Hi all, I have build testing cluster with 4 hosts, 1 SSD's and 11 HDD on each host. Running ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable) on Ubuntu. Because we want to save small size object, I set bluestore_min_alloc_size 8192 (it is maybe important in this case) I have filled it through rados gw with approx billion of small objects. After tests I changed min_alloc_size back and deleted rados pools (to emtpy whole cluster) and I was waiting till cluster deletes data from OSD's, but that destabilized the cluster. I never reached health OK. OSD's were killed in random order. I can start them back but they will again get out from cluster with.. ``` -18> 2020-09-05 22:11:19.430 7f7a3ee40700 5 prioritycache tune_memory target: 3221225472 mapped: 2064359424 unmapped: 8708096 heap: 2073067520 old mem: 1932735282 new mem: 1932735282 -17> 2020-09-05 22:11:19.430 7f7a3ee40700 5 bluestore.MempoolThread(0x555a9d0efb70) _trim_shards cache_size: 1932735282 kv_alloc: 1644167168 kv_used: 1644135504 meta_alloc: 142606336 meta_used: 143595 data_alloc: 142606336 data_used: 98304 -16> 2020-09-05 22:11:20.434 7f7a3ee40700 5 prioritycache tune_memory target: 3221225472 mapped: 2064941056 unmapped: 8126464 heap: 2073067520 old mem: 1932735282 new mem: 1932735282 -15> 2020-09-05 22:11:21.434 7f7a3ee40700 5 prioritycache tune_memory target: 3221225472 mapped: 2064359424 unmapped: 8708096 heap: 2073067520 old mem: 1932735282 new mem: 1932735282 -14> 2020-09-05 22:11:22.258 7f7a2b81f700 5 osd.42 103257 heartbeat osd_stat(store_statfs(0x1ce18290000/0x2d08c0000/0x1d180000000, data 0x23143355/0x974a0000, compress 0x0/0x0/0x0, omap 0x1f11e, meta 0x2d08a0ee2), peers [3,4,6,7,8,11,12,13,14,16,17,18,19,21,23,24,25,27,28,29,31,32,33,34,41,43] op hist []) -13> 2020-09-05 22:11:22.438 7f7a3ee40700 5 prioritycache tune_memory target: 3221225472 mapped: 2064359424 unmapped: 8708096 heap: 2073067520 old mem: 1932735282 new mem: 1932735282 -12> 2020-09-05 22:11:23.442 7f7a3ee40700 5 prioritycache tune_memory target: 3221225472 mapped: 2064359424 unmapped: 8708096 heap: 2073067520 old mem: 1932735282 new mem: 1932735282 -11> 2020-09-05 22:11:24.442 7f7a3ee40700 5 prioritycache tune_memory target: 3221225472 mapped: 2064285696 unmapped: 8781824 heap: 2073067520 old mem: 1932735282 new mem: 1932735282 -10> 2020-09-05 22:11:24.442 7f7a3ee40700 5 bluestore.MempoolThread(0x555a9d0efb70) _trim_shards cache_size: 1932735282 kv_alloc: 1644167168 kv_used: 1644119840 meta_alloc: 142606336 meta_used: 143595 data_alloc: 142606336 data_used: 98304 -9> 2020-09-05 22:11:24.442 7f7a2e024700 0 bluestore(/var/lib/ceph/osd/ceph-42) log_latency_fn slow operation observed for _collection_list, latency = 151.113s, lat = 2m cid =5.47_head start #5:e2000000::::0# end #MAX# max 2147483647 -8> 2020-09-05 22:11:24.446 7f7a2e024700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f7a2e024700' had timed out after 15 -7> 2020-09-05 22:11:24.446 7f7a2e024700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f7a2e024700' had suicide timed out after 150 -6> 2020-09-05 22:11:24.446 7f7a4c2a4700 10 monclient: get_auth_request con 0x555b15d07680 auth_method 0 -5> 2020-09-05 22:11:24.446 7f7a3c494700 2 osd.42 103257 ms_handle_reset con 0x555b15963600 session 0x555a9f9d6d00 -4> 2020-09-05 22:11:24.446 7f7a3c494700 2 osd.42 103257 ms_handle_reset con 0x555b15961b00 session 0x555a9f9d7980 -3> 2020-09-05 22:11:24.446 7f7a3c494700 2 osd.42 103257 ms_handle_reset con 0x555b15963a80 session 0x555a9f9d6a80 -2> 2020-09-05 22:11:24.446 7f7a3c494700 2 osd.42 103257 ms_handle_reset con 0x555b15960480 session 0x555a9f9d6f80 -1> 2020-09-05 22:11:24.446 7f7a3c494700 3 osd.42 103257 handle_osd_map epochs [103258,103259], i have 103257, src has [83902,103259] 0> 2020-09-05 22:11:24.450 7f7a2e024700 -1 *** Caught signal (Aborted) ** ``` I have approx 12 OSD's down with this error. I decided to wipe problematic OSD's so I cannot debug it, but I'm curious what I did wrong (deleting pool with many small data?) or what to do next time. I did that before but not with bilion object and without bluestore_min_alloc_size change, and it worked without problems. With regards Jan Pekar -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 http://www.imatic.cz | +420326555326 ============ --

2 3

cephadm - How to deploy ceph cluster with a partition on SSD for block.db
by klemen＠psi-net.si 11 Sep '20

11 Sep '20

I'm trying to deploy a ceph cluster with a cephadm tool. I've already successfully done all steps except adding OSDs. My testing equipment consists of three hosts. Each host has SSD storage, where OS is installed into. On that storage I created partition, which can be used as a ceph block.db. Hosts have also 2 additional HDs (spinning drives) for OSD data. On docs I couldn't find how to deploy such configuration. Do you have any hints, how to do that? Thanks for help!

5 7

slow "rados ls"
by Marcel Kuiper 11 Sep '20

11 Sep '20

Hi One of my clusters running nautilus 14.2.8 is very slow (13 seconds or so where my other clusters are returning almost instantanious) when doing a 'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors. I checked - ceph status => OK - routing to/from osds ok (I see a lot of established connections to osds due to the command, nothing in syn_sent indicating incomplete handshake) - ping times are OK - no interface errors - no packet drops - no increasing send queus - and as far as I can see nothing out of the ordinary in mon and osd logs I have no clue how to debug the issue. If someone has pointers it would be much appreciated Kind Regards Marcel

4 16

Re: ceph-osd performance on ram disk
by George Shuklin 11 Sep '20

11 Sep '20

Thank you! I know that article, but they promise 6 core use per OSD, and I got barely over three, and all this in totally synthetic environment with no SDD to blame (brd is more than fast and have a very consistent latency under any kind of load). On Thu, Sep 10, 2020, 19:39 Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > Hi George, > > Very interesting and also a bit expecting result. Some messages posted > here are already indicating that getting expensive top of the line > hardware does not really result in any performance increase above some > level. Vitaliy has documented something similar[1] > > [1] > https://yourcmc.ru/wiki/Ceph_performance > > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] ceph-osd performance on ram disk > > I'm creating a benchmark suite for Сeph. > > During benchmarking of benchmark, I've checked how fast ceph-osd works. > I decided to skip all 'SSD mess' and use brd (block ram disk, modprobe > brd) as underlying storage. Brd itself can yield up to 2.7Mpps in fio. > In single thread mode (iodepth=1) it can yield up to 750k IOPS. LVM over > brd gives about 600kIOPS in single-threaded mode with iodepth=1 (16us > latency). > > But, as soon as I put ceph-osd (bluestore) on it, I see something very > odd. No matter how much parallel load I push onto this OSD, it never > gives more than 30 kIOPS, and I can't understand where bottleneck is. > > CPU utilization: ~300%. There are 8 cores on my setup, so, CPU is not a > bottleneck. > > Network: I've moved benchmark on the same host as OSD, so it's a > localhost. Even counting network, it's still far away from saturation. > 30kIOPS (4k) is about 1Gb/s, but I have 10G links. Anyway, tests are run > on localhost, so network is irrelevant (I've checked it, traffic is on > localhost). Test itself consumes about 70% CPU of one core, so there are > plenty left. > > Replication: I've killed it (size=1, single osd in the pool). > > single-threaded latency: 200us, 4.8kIOPS. > iopdeth=32: 2ms (15kIOPS). > iodepth=16,numjobs=8: 5ms (24k IOPS) > > I'm running fio with 'rados' ioengine, and it looks like putting more > workers doesn't change much, so it's not rados ioengine. > > As there is plenty CPU and IO left, there is only one possible place for > bottleneck: some time-consuming single-threaded code in ceph-osd. > > Are there any knobs to tweak to see higher performance for ceph-osd? I'm > pretty sure it's not any kind of leveling, GC or other 'iops-related' > issues (brd has performance of two order of magnitude higher). > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >

3 6

Printer is in error state because of motherboard malfunction? Contact customer care.
by mary smith 11 Sep '20

11 Sep '20

Sometimes the motherboard glitch can cause printer is in error state situation. Therefore, you can use the troubleshooting solutions that are available in the tech consultancy websites or you can call the help team and use their assistance to deal with the issue. In addition to that, you can always watch some tech trick videos on Youtube. https://www.epsonprintersupportpro.net/epson-printer-in-error-state/

1 0

Errror in Facebook drafts? Find support by dialing Facebook Customer Service Toll Free Number.
by mary smith 11 Sep '20

11 Sep '20

The drafts are the place the message gets set aside in the occasion that you've excluded the recipient. In any case, if there's a glitch there, by then you can get it cured by using the help from the tech help destinations or you can dial the Facebook Customer Service Toll Free Number and have a conversation with the rep of the help gathering. https://www.customercare-email.com/facebook-customer-service.html

1 0

ceph config dump question
by Dave Baukus 10 Sep '20

10 Sep '20

A naive ceph user asks: I have 3 node cluster configured with 72 bluestore OSDs running on Ubuntu 20.01, Ceph Octopus 15.2.4 The cluster is configured via ceph-ansible stable-5.0. No configuration changes have been made outside of what is generated by ceph-ansible. I expected "ceph config dump" to show the entire configuration: # ceph config dump WHO MASK LEVEL OPTION VALUE RO # echo $? 0 Is my expectation wrong or is something broken ? If I set a value via ceph then it is reflected in the "ceph config dump" output # ceph config set osd osd_memory_target 2147483648 # ceph config dump WHO MASK LEVEL OPTION VALUE RO osd basic osd_memory_target 2147483648 If my expectation is wrong then how does one view the configuration defaults and all ? Thanks, -- Dave Baukus

1 0

Multipart uploads with partsizes larger than 16MiB failing on Nautilus
by shubjero 10 Sep '20

10 Sep '20

Hey all, I'm creating a new post for this issue as we've narrowed the problem down to a partsize limitation on multipart upload. We have discovered that in our production Nautilus (14.2.11) cluster and our lab Nautilus (14.2.10) cluster that multipart uploads with a configured part size of greater than 16777216 bytes (16MiB) will return a status 500 / internal server error from radosgw. So far I have increased the following rgw settings/values that looked suspect, without any success/improvement with partsizes. Such as: "rgw_get_obj_window_size": "16777216", "rgw_put_obj_min_window_size": "16777216", I am trying to determine if this is because of a conservative default setting somewhere that I don't know about or if this is perhaps a bug? I would appreciate it if someone on Nautilus with rgw could also test / provide feedback. It's very easy to reproduce and configuring your partsize with aws2cli requires you to put the following in your aws 'config' s3 = multipart_chunksize = 32MB rgw server logs during a failed multipart upload (32MB chunk/partsize): 2020-09-08 15:59:36.054 7f2d32fa6700 1 ====== starting new request req=0x55953dc36930 ===== 2020-09-08 15:59:36.082 7f2d32fa6700 -1 res_query() failed 2020-09-08 15:59:36.138 7f2d32fa6700 1 ====== req done req=0x55953dc36930 op status=0 http_status=200 latency=0.0839988s ====== 2020-09-08 16:00:07.285 7f2d3dfbc700 1 ====== starting new request req=0x55953dc36930 ===== 2020-09-08 16:00:07.285 7f2d3dfbc700 -1 res_query() failed 2020-09-08 16:00:07.353 7f2d00741700 1 ====== starting new request req=0x55954dd5e930 ===== 2020-09-08 16:00:07.357 7f2d00741700 -1 res_query() failed 2020-09-08 16:00:07.413 7f2cc56cb700 1 ====== starting new request req=0x55953dc02930 ===== 2020-09-08 16:00:07.417 7f2cc56cb700 -1 res_query() failed 2020-09-08 16:00:07.473 7f2cb26a5700 1 ====== starting new request req=0x5595426f6930 ===== 2020-09-08 16:00:07.473 7f2cb26a5700 -1 res_query() failed 2020-09-08 16:00:09.465 7f2d3dfbc700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.465 7f2d3dfbc700 1 ====== req done req=0x55953dc36930 op status=-35 http_status=500 latency=2.17997s ====== 2020-09-08 16:00:09.549 7f2d00741700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.549 7f2d00741700 1 ====== req done req=0x55954dd5e930 op status=-35 http_status=500 latency=2.19597s ====== 2020-09-08 16:00:09.605 7f2cc56cb700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.609 7f2cc56cb700 1 ====== req done req=0x55953dc02930 op status=-35 http_status=500 latency=2.19597s ====== 2020-09-08 16:00:09.641 7f2cb26a5700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.641 7f2cb26a5700 1 ====== req done req=0x5595426f6930 op status=-35 http_status=500 latency=2.16797s ====== awscli client side output during a failed multipart upload: root@jump:~# aws --no-verify-ssl --endpoint-url http://lab-object.cancercollaboratory.org:7480 s3 cp 4GBfile s3://troubleshooting upload failed: ./4GBfile to s3://troubleshooting/4GBfile An error occurred (UnknownError) when calling the UploadPart operation (reached max retries: 2): Unknown Thanks, Jared Baker Cloud Architect for the Cancer Genome Collaboratory Ontario Institute for Cancer Research

3 6

Octopus dashboard: rbd-mirror page shows error for primary site
by Eugen Block 10 Sep '20

10 Sep '20

Hi *, I was just testing rbd-mirror on ceph version 15.2.4-864-g0f510cb110 (0f510cb1101879a5941dfa1fa824bf97db6c3d08) octopus (stable) and noticed mgr errors on the primary site (also in version 15.2.2): ---snip--- 2020-09-10T11:20:01.724+0200 7f1c1b46a700 0 [dashboard ERROR controllers.rbd_mirror] Failed to list mirror image status rbd-pool1 Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in _get_pool_datum for image in mirror_image_status File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in <listcomp> for image in mirror_image_status KeyError: 'description' 2020-09-10T11:20:01.724+0200 7f1c1b46a700 0 [dashboard ERROR viewcache] Error while calling fn=<function _get_pool_datum at 0x7f1c5ac58f28> ex='description' Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/tools.py", line 156, in run val = self.fn(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in _get_pool_datum for image in mirror_image_status File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in <listcomp> for image in mirror_image_status KeyError: 'description' 2020-09-10T11:20:01.724+0200 7f1c1ec71700 0 [dashboard ERROR viewcache] Error while calling fn=<function _get_content_data at 0x7f1c5ac610d0> ex='description' Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/tools.py", line 156, in run val = self.fn(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 283, in _get_content_data _, pool = _get_pool_datum(pool_name) File "/usr/share/ceph/mgr/dashboard/tools.py", line 254, in wrapper return rvc.run(fn, args, kwargs) File "/usr/share/ceph/mgr/dashboard/tools.py", line 236, in run raise self.exception File "/usr/share/ceph/mgr/dashboard/tools.py", line 156, in run val = self.fn(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in _get_pool_datum for image in mirror_image_status File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in <listcomp> for image in mirror_image_status KeyError: 'description' 2020-09-10T11:20:01.724+0200 7f1c23ebb700 0 [dashboard ERROR request] [X.X.X.X:42050] [GET] [500] [0.020s] [admin] [513.0B] /api/block/mirroring/summary 2020-09-10T11:20:01.728+0200 7f1c23ebb700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "64ea5407-171b-4f6c-862e-fc881e68fec0"} ---snip--- The mirroring itself seems to work fine, I haven't noticed issues with the image on the secondary site. I also uploaded a screenshot of the dashboard [1], the installation is on openSUSE Leap 15.2. I couldn't find a tracker issue or a report in bugzilla.opensuse.org, but before creating a ticket I wanted to ask if this is a known issue. The rbd-mirror panel works fine on the secondary site, I assume because there is actually an rbd-mirror daemon running. Or maybe I just misunderstand the purpose, but if there is no mirror daemon on site A there shouldn't be an error message, should it? Regards, Eugen [1] https://paste.opensuse.org/36442780

2 3

Octopus: snapshot errors during rbd import
by Eugen Block 10 Sep '20

10 Sep '20

Hi *, I'm currently testing rbd-mirror on ceph version 15.2.4-864-g0f510cb110 (0f510cb1101879a5941dfa1fa824bf97db6c3d08) octopus (stable) and saw this during an rbd import of a fresh image on the primary site: ---snip--- ceph1:~ # rbd import /mnt/SUSE-OPENSTACK-CLOUD-7-x86_64-GM-DVD1.iso rbd-pool1/cloud7 Importing image: 5% complete...2020-09-10T12:00:10.220+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c022060 send_op: snapshot doesn't exist 2020-09-10T12:00:10.224+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c022060 should_complete: encountered error: (2) No such file or directory 2020-09-10T12:00:10.232+0200 7ff1ba7fc700 -1 librbd::Operations: No such snapshot found. Importing image: 6% complete...2020-09-10T12:00:13.940+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c032eb0 send_op: snapshot doesn't exist 2020-09-10T12:00:13.940+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c032eb0 should_complete: encountered error: (2) No such file or directory Importing image: 7% complete...2020-09-10T12:00:18.596+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c027af0 send_op: snapshot doesn't exist 2020-09-10T12:00:18.596+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c027af0 should_complete: encountered error: (2) No such file or directory Importing image: 8% complete...2020-09-10T12:00:24.048+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c02f940 send_op: snapshot doesn't exist 2020-09-10T12:00:24.048+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c02f940 should_complete: encountered error: (2) No such file or directory Importing image: 100% complete...done. ---snip--- The import eventually succeeds and the image is fine (md5sum on the remote image is correct), but I'm wondering what snapshot it is looking for since there aren't any (yet). Here's some info about the image (not sure if it's relevant): ---snip--- ceph1:~ # rbd info rbd-pool1/cloud7 rbd image 'cloud7': size 1.4 GiB in 347 objects order 22 (4 MiB objects) snapshot_count: 0 id: 149e4eb8e387a block_name_prefix: rbd_data.149e4eb8e387a format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Thu Sep 10 11:59:18 2020 access_timestamp: Thu Sep 10 11:59:18 2020 modify_timestamp: Thu Sep 10 12:03:30 2020 journal: 149e4eb8e387a mirroring state: enabled mirroring mode: journal mirroring global id: 82bb30ec-bd33-4f84-9d0c-b3650b567670 mirroring primary: true ---snip--- Is this worth a bug report or is it already known and I just couldn't find anything? Regards, Eugen

2 2

Re: Moving OSD from one node to another
by Eugen Block 10 Sep '20

10 Sep '20

Hi, I haven't done this myself yet but you should be able to simply move the (virtual) disk to the new host and start the OSD, depending on the actual setup. If those are stand-alone OSDs (no separate DB/WAL) it shouldn't be too difficult [1]. If you're using ceph-volume you could run 'ceph-volume lvm trigger' in case the OSD doesn't restart. Although this would not reconstruct data it would start a rebalance process since the crushmap will change automatically, leading to misplaced objects. Regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-March/025624.html Zitat von huxiaoyu(a)horebdata.cn: > Dear ceph folks, > > I encoutered an interesting situation as follows: an old FC SAN is > connected two ceph OSD nodes, and its LUNs are used as virtual OSDs. > When one node fails, its LUN can be taken over by anther node. My > question is, how to start up the OSD on the new node without > reconstructing its data? In other words, is there a simple to move > OSD from one node to another? > > many thanks, > > samuel > > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Moving OSD from one node to another
by huxiaoyu＠horebdata.cn 09 Sep '20

09 Sep '20

Dear ceph folks, I encoutered an interesting situation as follows: an old FC SAN is connected two ceph OSD nodes, and its LUNs are used as virtual OSDs. When one node fails, its LUN can be taken over by anther node. My question is, how to start up the OSD on the new node without reconstructing its data? In other words, is there a simple to move OSD from one node to another? many thanks, samuel huxiaoyu(a)horebdata.cn

1 0

Storage class usage stats
by Tobias Urdin 09 Sep '20

09 Sep '20

Hello, I've been trying to understand if there is any way to get usage information based on storage classes for buckets. Since there is no information available from the "radosgw-admin bucket stats" commands nor any other endpoint I tried to browse the source code but couldn't find any references where the storage class would be exposed in such a way. It also seems that RadosGW today is not saving any counters on amount of objects stored in storage classes when it's collecting usage stats, which means there is no such metadata saved for a bucket. I was hoping it was atleast saved but not exposed because then it would have been a easier fix than adding support to count number of objects in storage classes based on operations which would involve a lot of places and mean writing to the bucket metadata on each op :( Is my assumptions correct that there is no way to retrieve such information, meaning there is no way to measure such usage? If the answer is yes, I assume the only way to get something that could be measured would be to instead have multiple placement targets since that is exposed from in bucket info. The bad things would be though that you lose a lot of functionality related to lifecycle and moving a single object to another storage class. Best regards Tobias

2 2

OSDs and tmpfs
by Shain Miley 09 Sep '20

09 Sep '20

Hi, I recently added 3 new servers to Ceph cluster. These servers use the H740p mini raid card and I had to install the HWE kernel in Ubuntu 16.04 in order to get the drives recognized. We have a 23 node cluster and normally when we add OSDs they end up mounting like this: /dev/sde1 3.7T 2.0T 1.8T 54% /var/lib/ceph/osd/ceph-15 /dev/sdj1 3.7T 2.0T 1.7T 55% /var/lib/ceph/osd/ceph-20 /dev/sdd1 3.7T 2.1T 1.6T 58% /var/lib/ceph/osd/ceph-14 /dev/sdc1 3.7T 1.8T 1.9T 49% /var/lib/ceph/osd/ceph-13 However I noticed this morning that the 3 new servers have the OSDs mounted like this: tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-246 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-240 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-248 tmpfs 47G 28K 47G 1% /var/lib/ceph/osd/ceph-237 Is this normal for deployments going forward…or did something go wrong? These are 12TB drives but they are showing up as 47G here instead. We are using ceph version 12.2.13 and I installed this using ceph-deply version 2.0.1. Thanks in advance, Shain Shain Miley | Director of Platform and Infrastructure | Digital Media | smiley(a)npr.org

1 0

Cleanup orphan osd process in octopus
by levindecaro＠gmail.com 09 Sep '20

09 Sep '20

Hi there, I got a handicap problem on ceph orch which cannot found anywhere from doc to solve this problem, I was removed an osd using ceph purge osd.27 (old style) and forgotten there are new way in Octopus. Now i got two osd process via ceph orch ps, is it have a way to clean it up properly? osd.27 sds-0 running (7h) 49s ago 7h 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de acab3229c945 osd.27 sds-2 stopped 19s ago 4d <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown>

1 0

How to delete OSD benchmark data
by Jayesh Labade 09 Sep '20

09 Sep '20

Dear Ceph Users, I am testing my 3 node Proxmox + Ceph cluster. I have performed osd benchmark with the below command. # ceph tell osd.0 bench Do I need to perform any cleanup to delete benchmark data from osd ? I have googled for same but nowhere mentioned post steps after osd benchmark commands Thanks Jayesh

2 2

RGW bucket sync
by Eugen Block 09 Sep '20

09 Sep '20

Hi *, I'm wondering about what actually happens in the ceph cluster if I copy/sync the content of one bucket into a different bucket. I'll just describe what I saw and maybe someone could clarify what is happening. I have a RGW in a small test cluster (15.2.2) and created a bucket (bucket1) with s3cmd, then put a large file into bucket1. This takes some time, of course, I see the network utilization on the client and also the OSD load on the cluster during the upload (expected). Then I create bucket2 and run 's3cmd cp s3://bucket1/file s3://bucket2' which takes only a couple of seconds. I only see some OSD load for a short period of time during the copy process, but that's it. The copied file is available almost immediately. How does this work? It seems as if there's (almost) no client traffic (except for the cp command, of course) to recreate the file in the second bucket, as if the OSDs are directly instructed to create copies of the objects. I would highly appreciate it if anyone could clarify this for me. Thanks! Eugen

3 4

How to working with ceph octopus multisite-sync-policy
by system.engineer.mon＠gmail.com 09 Sep '20

09 Sep '20

Dear All I build 2 ceph poc cluster My ceph version is octopus 15.2.4 with centos 8 I configure ceph multisite is working Cluster_1 realm 20f3a458-b0d4-4699-b8f7-75411366c635 (global) zonegroup 94b70bdb-1492-49b7-8cda-210d3552ecd2 (data) zone bb4106bc-6a01-44e7-be53-ce8c5dbbce97 (hy-1) metadata sync no sync (zone is master) data sync source: 36373df5-bcd8-4eea-954b-3b6158d57c80 (hy-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Cluster_2 realm 20f3a458-b0d4-4699-b8f7-75411366c635 (global) zonegroup 94b70bdb-1492-49b7-8cda-210d3552ecd2 (data) zone 36373df5-bcd8-4eea-954b-3b6158d57c80 (hy-2) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: bb4106bc-6a01-44e7-be53-ce8c5dbbce97 (hy-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source 2 Cluster is Active/Active Create bucket in cluster_1 => It automatic sync to cluster_2 Create bucket in cluster_2 => It automatic sync to cluster_1 Put data in bucket cluster_1 => It automatic sync to bucket in cluster_2 Put data in bucket cluster_2 => It automatic sync to bucket in cluster_1 I try configure example_3 and example_4 but not working https://docs.ceph.com/docs/master/radosgw/multisite-sync-policy/ Anyone, Can help or recommend me ? Best Regrads, Engineer_Mon

1 0

Problem with /etc/ceph/iscsi-gateway.cfg checksum
by Matthias Leopold 09 Sep '20

09 Sep '20

Hi, I'm preparing three Ceph Nautilus nodes as iSCSI gateways following this documentation: https://docs.ceph.com/docs/nautilus//rbd/iscsi-overview/. For some reason rbd-target-api for one node doesn't agree on /etc/ceph/iscsi-gateway.cfg although "sha256sum /etc/ceph/iscsi-gateway.cfg" is the same: ~$ ssh -l root tentakel4 "sha256sum /etc/ceph/iscsi-gateway.cfg" e65444a15bbe9a115fd27040233883b681226e922367f7c61f7378d596ed12fd /etc/ceph/iscsi-gateway.cfg ~$ ssh -l root tentakel5 "sha256sum /etc/ceph/iscsi-gateway.cfg" e65444a15bbe9a115fd27040233883b681226e922367f7c61f7378d596ed12fd /etc/ceph/iscsi-gateway.cfg ~$ ssh -l root tentakel6 "sha256sum /etc/ceph/iscsi-gateway.cfg" e65444a15bbe9a115fd27040233883b681226e922367f7c61f7378d596ed12fd /etc/ceph/iscsi-gateway.cfg ~$ curl --user foo:bar -X GET http://tentakel4:5000/api/sysinfo/checkconf { "data": "110a32477ca026225bceab337535e0cad522d8ea10e69922e1ef9412331b8d94" } ~$ curl --user foo:bar -X GET http://tentakel5:5000/api/sysinfo/checkconf { "data": "faee0b958ea8f1bcbfaae4a40ca247af2271d439b54fa345be5a3fa2283dcc84" } ~$ curl --user foo:bar -X GET http://tentakel6:5000/api/sysinfo/checkconf { "data": "110a32477ca026225bceab337535e0cad522d8ea10e69922e1ef9412331b8d94" } On tentakel5 I'm constantly seeing "INFO [rbd-target-api:2821:run()] - Change detected - internal 0 / xattr 21 refreshing" in /var/log/rbd-target-api/rbd-target-api.log Can somebody give me a hint how to fix this? Software: - CentOS 7.8 - ceph-iscsi-3.4-1.el7.noarch - tcmu-runner-1.5.2-1.el7.x86_64 - sestatus: permissive thx Matthias

1 0

Error in OS causing Epson Error Code 0x97 pop up? Get to assistance.
by mary smith 09 Sep '20

09 Sep '20

The OS can have some compatibility issues with the printer and that might result in Epson Error Code 0x97. Therefore, to deal with the issue you can use the help steps that are mentioned in the tech consultancy sites or you can also get the issue resolved by using the assistance that is available in the tech videos. https://www.epsonprintersupportpro.net/epson-error-code-0x97/

1 0

cephadm didn't create journals
by Darrin Hodges 08 Sep '20

08 Sep '20

HI all, I have created a new cephs cluster using the cephadm command and it appears to work very well. I tried to specify using an ssd for the journals but it doesn't appear to have worked. My yaml file is: service_type: osd service_id: default_drive_group placement: host_pattern: 'ceph-osd0[1-4]' journal_devices: rotational: 0 data_devices: rotational: 1 encrypted: true all of the osd's were created but it appears the journals are on each disk rather than the ssd. Can I add journals after the fact with cephadm ? Are they even still a thing? thanks Darrin -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

3 2

10 Helpful Applications for Students
by elianajones117＠gmail.com 08 Sep '20

08 Sep '20

Here is the rundown of the best applications accessible on Google Play Store and App store for the understudy:- 1. Note Taking 2. Writing Paper 3. Screen Recorder 4. Study Groups 5. Programming 6. Health 7. Presentations 8. Time Management 9. Math 10. Career Tools Regards: [url=http://uscomoffice.com/setup/]Office.com/Setup[/url] | [url=http://go-mcafeecomactivate.com/]McAfee.com/Activate[/url]

4 4

how to write a good essay
by 27nick1993＠gmail.com 08 Sep '20

08 Sep '20

Are you busy and do not have time to handle your assignment? Are you scared that your paper will not make the grade? Do you have responsibilities that may hinder you from turning in your assignment on time? Are you tired and can barely handle your assignment? Are your grades inconsistent? Whichever your reason may is, it is valid! You can get professional academic help from our service https://helpinhomework.org/essay-writing-services at affordable rates. We have a team of professional academic writers who can handle all your assignments. Why choose our academic writing service? Plagiarism free papers Timely delivery Any deadline Skilled, Experienced Native English Writers Subject-relevant academic writer Adherence to paper instructions Ability to tackle bulk assignments Reasonable prices 24/7 Customer Support Get superb grades consistently Post your assignment Now on https://helpinhomework.org/essay-writing-services

1 0

Release mental stress using Hong Kong Assignment help
by james wick 08 Sep '20

08 Sep '20

Many students can’t complete their assignments within the assigned date because of some unavoidable circumstances in Hong Kong. It’s true that your mental stress raises hindrance and disturbs your concentration for your work. Because of stress and tension, you will not have enough thoughts to make your work effective. This situation directs you to the platform of Assignment Help services even in Hong Kong. This is because an unstable mind could not generate the right ideas to compose a worthy assignment. Make your assignment informative and productive using the online writing services of assignment experts. Professionals have a better understanding of the subject and know how to frame all information in the right format. So, you can use online assignment help when you have issues in composing your academic papers irrespective of any subject. https://www.greatassignmenthelp.com/hk/

4 3

How to deal with "inconsistent+failed_repair" pgs on cephfs pool ?
by Kári Bertilsson 08 Sep '20

08 Sep '20

Hello I have a couple of pgs with inconsitent+failed_repair after trying "ceph pg repair" on them. All the OSD's are using bluestore. These pgs belong to an erasure coded pool that runs cephfs. I did manage to find details about which objects are inconsistent by using the "rados list-inconsistent-obj pgid --format=json-pretty" But I am wondering how to find which files the corrupt objects belong to ? Also since I am guessing there is no way to recover these objects i will need to delete them to get back to active+clean status. What is the best way to delete the corrupt objects ? Best regards, Kári

1 0

Messenger v2 and IPv6-only still seems to prefer IPv4 (OSDs stuck in booting state)
by Wido den Hollander 07 Sep '20

07 Sep '20

Hi, Last night I've spend a couple of hours debugging a issue where OSDs would be marked as 'up', but then PGs stayed in the 'peering' state. Looking through the admin socket I saw these OSDs were in the 'booting' state. Looking at the OSDMap I saw this: osd.3 up in weight 1 up_from 26 up_thru 700 down_at 0 last_clean_interval [0,0) [v2:[2a05:xx0:700:2::7]:6816/7923,v1:[2a05:xx:700:2::7]:6817/7923,v2:0.0.0.0:6818/7923,v1:0.0.0.0:6819/7923] [v2:[2a05:xx:700:2::7]:6820/7923,v1:[2a05:1500:700:2::7]:6821/7923,v2:0.0.0.0:6822/7923,v1:0.0.0.0:6823/7923] exists,up 786d3e9d-047f-4b09-b368-db9e8dc0805d In ceph.conf this was set: ms_bind_ipv6 = true public_addr = 2a05:xx:700:2::6 On true IPv6-only nodes this works fine. But on nodes where there is also IPv4 present this can (and will?) cause problems. It did not use tcpdump/wireshark to investigate, but it seems that the OSDs tried to contact each other. Using the 0.0.0.0 IPv4 address. After adding these settings the problems were resolved: ms_bind_msgr1 = false ms_bind_ipv4 = false This also disables msgrv1 as we didn't need it here. A cluster and clients all running Octopus. The OSDMap now showed: osd.3 up in weight 1 up_from 704 up_thru 712 down_at 702 last_clean_interval [26,701) v2:[2a05:xx:700:2::7]:6804/791503 v2:[2a05:xx:700:2::7]:6805/791503 exists,up 786d3e9d-047f-4b09-b368-db9e8dc0805d OSDs can back right away, PGs peered and the problems were resolved. Wido

2 1

Tips for Taking a Semester off From College
by serene.banks3＠gmail.com 07 Sep '20

07 Sep '20

For most students, faculty may be the top 4 or 5 decades in their lives. But, in addition, there are many students who get to school and begin to experience burnout or confusion on what they wish to do after college. Sometimes this may result in falling grades, or pupils questioning whether going to school was even the correct choice for them. In some specific situations, some individuals have even taken a semester off so as to recharge their batteries regain focus in their research. Even though this may be a good thing to do, it may also be a challenging choice to describe to parents, loved ones, friends, and fellow pupils. Listed here are tips/strategies for folks to make the most of their off time before going back to school to receive their level. In case you decide you wish to take some time off from college to clean your mind, you need to produce a plan to be able to market the idea for your parents. The strategy won't simply set their mind at ease, but additionally, it will be something which you can return on during the session to keep you on course. You should think about your strategy nearly just like a business program. Be certain you account for the motives which you're taking time off and everything you intend on doing during each period of the time off from college. It'll be an additional bonus if you're able to describe what sorts of lessons you'll be studying during every stage. Don't underestimate how important with a strategy could be. Parents may initially see your session off with stress since they will fear you will eliminate motivation to return to college. They generally have a substantial monetary investment tied to your academic achievement. And of course that the deficiency of a strategy will also cause an absence of private attention and inspiration on your own end. Obtain Work, Gain Expertise The very best thing you may do during your off time is get work. Whether it's paid or unpaid, awaken each morning and operate. The job doesn't need to be fulltime and if you're creative it doesn't even need to be working for someone else. There are loads of students who've tried their hands small start-up companies with varying levels of success. The most important goal regardless of what you are doing would be to obtain expertise and transferable skills which you can add to a resume. The occupation skills are not only going to help you build career-wise, but also give you a hand in interviews once it is possible to bet you will be asked the reason you chose off a semester and everything you did during this period of time. Your short term job might even allow you to concentrate on a profession or career path you had not considered before your fracture. Among the most difficult things to do throughout time away from faculty is to keep your brain self-evident. Though your classmates are ongoing to move to class, take examinations, and also write term papers, you'll be away in your grind doing something entirely different. To be able to keep on creating the intellectual portion of your mind, you need to keep on reading anything that's near you. Whether it's a paper, book, or something from the New York Times bestseller list, browse it. Whenever you're finished reading it, then attempt and find someone to talk about it with this it is possible to keep on practicing your critical thinking abilities. Do to do your very best to stay current with current affairs. View the news sometimes, or see websites like CNN to remain fresh on what's happening in the world. Another positive point to do would be create a reading list of publications inside your major. From the time you contact school, you'll be armed with knowledge and questions your classmates might not have . Preventing your intellectual side is only going to result in you needing to prevent return to college afterwards, which will only cause an overall demanding transition back to school. Among the gorgeous things about being at school is being subjected to various individuals from so many diverse areas. When you take off your time, it's very likely that you will return to exactly the identical region which you just grew up in, and also the exact very same thoughts to which you're constantly exposed previously. If you can, attempt to discover some opportunity to traveling to other regions throughout your break. It can wind up as a large trip to Europe or the Caribbean, or it might only be a trip from state to go to some relatives. In any circumstance, the excursion won't just keep you thrilled, but additionally, it will expand your horizons. Have a little time to take new areas of interest and speak to as many individuals as possible from the new places. In the end, it might never be as simple to travel as soon as you're back in school or the working world, take advantage of it. . .To Everybody During the time you're not at college, go out of the way to speak to as many individuals as you can. Speak with your parents. Speak with your pals. Speak with your loved ones. Speak with your classmates back in college. These new folks will let you break from your typical school social bubble, and also be exposed to new methods of believing. As tempting as it might be, don't sit home all day and play games or even watch tv. The very best approach to find out more about yourself is by interacting with different individuals in the external world. If you falling your semester maybe you need to use one of the official essay writing services like writepaperforme.review https://writepaperforme.review/

1 0

add debian buster stable support for ceph-deploy
by Jelle de Jong 07 Sep '20

07 Sep '20

Hello everybody, Can somebody add support for Debian buster and ceph-deploy: https://tracker.ceph.com/issues/42870 Highly appreciated, Regards, Jelle de Jong

6 8

Does Cash.app/help useful for discount?
by mary smith 07 Sep '20

07 Sep '20

Cash app to Cash app installment is normally moment and store effectively inside 10 seconds. Thus, you don't get a choice to drop the installment. On the off chance that there is any sum deducted from your record, at that point use Cash.app/help to get a discount. The sum appears for you when the sum is discounted by the beneficiary. https://www.cashapp-customerservice.com/

1 0

Recover pgs from failed osds
by Vahideh Alinouri 07 Sep '20

07 Sep '20

Ceph cluster is updated from nautilus to octopus. On ceph-osd nodes we have high I/O wait. After increasing one of pool’s pg_num from 64 to 128 according to warning message (more objects per pg), this lead to high cpu load and ram usage on ceph-osd nodes and finally crashed the whole cluster. Three osds, one on each host, stuck at down state (osd.34 osd.35 osd.40). Starting the down osd service causes high ram usage and cpu load and ceph-osd node to crash until the osd service fails. The active mgr service on each mon host will crash after consuming almost all available ram on the physical hosts. I need to recover pgs and solving corruption. How can i recover unknown and down pgs? Is there any way to starting up failed osd? Below steps are done: 1- osd nodes’ kernel was upgraded to 5.4.2 before ceph cluster upgrading. Reverting to previous kernel 4.2.1 is tested for iowate decreasing, but it had no effect. 2- Recovering 11 pgs on failed osds by export them using ceph-objectstore-tools utility and import them on other osds. The result followed: 9 pgs are “down” and 2 pgs are “unknown”. 2-1) 9 pgs export and import successfully but status is “down” because of "peering_blocked_by" 3 failed osds. I cannot lost osds because of preventing unknown pgs from getting lost. pgs size in K and M. "peering_blocked_by": [ { "osd": 34, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 35, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 40, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] 2-2) 1 pg (2.39) export and import successfully, but after starting osd service (pg import to it), ceph-osd node RAM and CPU consumption increase and cause ceph-osd node to crash until the osd service fails. Other osds become "down" on ceph-osd node. pg status is “unknown”. I cannot use "force-create-pg" because of data lost. pg 2.39 size is 19G. # ceph pg map 2.39 osdmap e40347 pg 2.39 (2.39) -> up [32,37] acting [32,37] # ceph pg 2.39 query Error ENOENT: i don't have pgid 2.39 *pg 2.39 info on failed osd: # ceph-objectstore-tool --data-path /var/lib/ceph/osd/*ceph-34* --op info --pgid 2.39 { "pgid": "2.39", "last_update": "35344'6456084", "last_complete": "35344'6456084", "log_tail": "35344'6453182", "last_user_version": 10595821, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 146, "epoch_pool_created": 79, "last_epoch_started": 25208, "last_interval_started": 25207, "last_epoch_clean": 25208, "last_interval_clean": 25207, "last_epoch_split": 370, "last_epoch_marked_full": 0, "same_up_since": 8347, "same_interval_since": 25207, "same_primary_since": 8321, "last_scrub": "35328'6440139", "last_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "last_deep_scrub": "35261'6031075", "last_deep_scrub_stamp": "2020-08-17T01:59:26.606037+0430", "last_clean_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "prior_readable_until_ub": 0 }, "stats": { "version": "35344'6456082", "reported_seq": "11733156", "reported_epoch": "35344", "state": "active+clean", "last_fresh": "2020-08-19T14:16:18.587435+0430", "last_change": "2020-08-19T12:00:59.377747+0430", "last_active": "2020-08-19T14:16:18.587435+0430", "last_peered": "2020-08-19T14:16:18.587435+0430", "last_clean": "2020-08-19T14:16:18.587435+0430", "last_became_active": "2020-08-06T00:23:51.016769+0430", "last_became_peered": "2020-08-06T00:23:51.016769+0430", "last_unstale": "2020-08-19T14:16:18.587435+0430", "last_undegraded": "2020-08-19T14:16:18.587435+0430", "last_fullsized": "2020-08-19T14:16:18.587435+0430", "mapping_epoch": 8347, "log_start": "35344'6453182", "ondisk_log_start": "35344'6453182", "created": 146, "last_epoch_clean": 25208, "parent": "0.0", "parent_split_bits": 7, "last_scrub": "35328'6440139", "last_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "last_deep_scrub": "35261'6031075", "last_deep_scrub_stamp": "2020-08-17T01:59:26.606037+0430", "last_clean_scrub_stamp": "2020-08-19T12:00:59.377593+0430", "log_size": 2900, "ondisk_log_size": 2900, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 19749578960, "num_objects": 2442, "num_object_clones": 20, "num_object_copies": 7326, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 2442, "num_whiteouts": 0, "num_read": 16120686, "num_read_kb": 82264126, "num_write": 19731882, "num_write_kb": 379030181, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 2861, "num_bytes_recovered": 21673259070, "num_keys_recovered": 32, "num_objects_omap": 2, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 152, "num_omap_keys": 16, "num_objects_repaired": 0 }, "up": [ 40, 35, 34 ], "acting": [ 40, 35, 34 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 40, "acting_primary": 40, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 25208, "hit_set_history": { "current_last_update": "0'0", "history": [] } } *pg 2.39 info on osd which import to it: # ceph-objectstore-tool --data-path /var/lib/ceph/osd/*ceph-37* --op info --pgid 2.39 PG '2.39' not found 2-3) 1 pg (2.79) is lost! This pg is not found on any of three failed osds (osd.34 osd.35 osd.40)! status is “unknown”. pg 2.79 export is failed: " PG '2.79' not found" # ceph pg map 2.79 Error ENOENT: i don't have pgid 2.79 # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-34 --op info --pgid 2.79 PG '2.79' not found 3- Using https://gitlab.lbader.de/kryptur/ceph-recovery/tree/master but it does not work for recent ceph versions and tested on “hammer” release. 4- Using https://ceph.io/planet/recovering-from-a-complete-node-failure/ but in lvm scenario I could not mount failed osd lv to new /var/lib/ceph/osd/ceph-x* .*Could not prepare and activate new osd to failed osd disk. 5- Setting pool min_size=1 that down pgs belong to it, restart osds that pgs import to them but no changes. 6- Seting pool min_size=1 that pg 2.39 belong to it, restart osds that pg import to them but no changes. 7- Repairing failed osds using ceph-objectstore-tools, making “in” and starting them but no changes. # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-x --op repair 8- Repairing 2 unknown pgs, but no changes. # ceph pg repaire 2.39 # ceph pg repair 2.79 9- Forcing recovery 2 unknown pgs, but no changes. # ceph pg force-recovery 2.39 # ceph pg force-recovery 2.79 10- Check PID count in ceph-osd nodes because of osd services failed to start. kernel.pid.max = 4194304 11- Raising osd_op_thread_suicide_timeout=900, but no change.

3 12

Re: PG number per OSD
by norman 07 Sep '20

07 Sep '20

The current OSD's size and pg_num? Are you using the different size OSDs? On 6/9/2020 上午1:34, huxiaoyu(a)horebdata.cn wrote: > Dear Ceph folks, > > As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? > > thanks a lot, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Can 16 server grade ssd's be slower then 60 hdds? (no extra journals)
by VELARTIS Philipp Dürhammer 06 Sep '20

06 Sep '20

I have a productive 60 osd's cluster. No extra Journals. Its performing okay. Now I added an extra ssd Pool with 16 Micron 5100 MAX. And the performance is little slower or equal to the 60 hdd pool. 4K random as also sequential reads. All on dedicated 2 times 10G Network. HDDS are still on filestore. SSD on bluestore. Ceph Luminous. What should be possible 16 ssd's vs. 60 hhd's no extra journals?

6 16

damaged cephfs
by Magnus HAGDORN 06 Sep '20

06 Sep '20

Hi there, we reconfigured our ceph cluster yesterday to remove the cluster network and things didn't quite go to plan. I am trying to figure out what went wrong and also what to do next. We are running nautilus 14.2.10 on Scientific Linux 7.8. So, we are using a mixture of RBDs and cephfs. For the transition we switched off all machines that are using the RBDs and switched off the cephfs using ceph fs set one down true Once no more MDS were running we reconfigured ceph to remove the cluster network and set various flags ceph osd set noout ceph osd set nodown ceph osd set pause ceph osd set nobackfill ceph osd set norebalance ceph osd set norecover We then restarted the OSDs one host at a time. During this process ceph was mostly happy, except for two PGs. After all OSDs had been restarted we switched off the cluster network switches to make sure it was totally gone. ceph was still happy. The PG error also disappeared. We then unset all those errors and re-enabled cephfs. We then switched on the servers using the RBDs with no issues. So far so good. We then started using the cephfs (we keep VM images on the cephfs). The MDS were showing an error. I restarted the MDS but they didn't come back. We then followed the instructions here: https://docs.ceph.com/docs/nautilus/cephfs/disaster-recovery-experts/#disas… up to truncating the journal. The MDS started again. However, as soon as we started writing the cephfs the MDS crashed. A scrub of the cephfs revealed backtrace damage. We have now followed the remaining steps of the disaster recovery procedure and are waiting for the cephfs-data-scan scan_extents to complete. It would be really helpful if you could give an indication of how long this process will take (we have ~40TB in our cephfs) and how many workers to use. The other missing bit of documentation is the cephfs scrubbing. Is that something we should run routinely? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 3

ceph fs reset situation
by Alexander B. Ustinov 06 Sep '20

06 Sep '20

Hello, list. Have anybody been in the situation when after "ceph fs reset" filesystem becomes blank (mounts OK, ls shows no files/directories), but data and metadata pools still hold something (698G and 400M respectively by "ceph fs status"). Would be grateful for documentation vectors and/or suggestions. Maybe i remember wrong, but few times in the past same "ceph fs reset" produced minor corruption to recent filesystem changes.

1 0

Re: PG number per OSD
by Anthony D'Atri 06 Sep '20

06 Sep '20

I think there are multiple variables there. My advice is for HDDs to aim for an average of 150-200 as I wrote before. The limitation is the speed of the device, throw a thousand PGs on there and you won’t get any more out of it, you’ll just have more peering and more RAM used. NVMe is a different story. > > Is there any rules for computing RAM requeirements in terms of the number of PGs? > > Just curious abount what is the fundamental limitations on the number of PGs per OSD for bigger capacity HDD > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn > > From: Anthony D'Atri > Date: 2020-09-05 20:00 > To: huxiaoyu(a)horebdata.cn > CC: ceph-users > Subject: Re: [ceph-users] PG number per OSD > One factor is RAM usage, that was IIRC the motivation for the lowering of the recommendation of the ratio from 200 to 100. Memory needs also increase during recovery and backfill. > > When calculating, be sure to consider repllicas. > > ratio = (pgp_num x replication) / num_osds > > As HDDs grow the interface though isn’t becoming faster (with SATA at least), and there are only so many IOPS and MB/s that you’re going to get out of one no matter how you slice it. Everything always depends on your use-case and workload, but I suspect that often the bottleneck is the drive, not PG or OSD serialization. > > For example, do you prize IOPS more, latency, or MB/s? If you don’t care about latency, then you can drive your HDDs harder and get more MB/s throughput out of them, though your average latency might climb to 100ms. Which eg. RBD VM clients probably wouldn’t be too happy about, but which an object service *might* tolerate. > > Basically in the absence of more info, I would personally suggest aiming at the 150-200 average range, with pgp_num a power of 2. If you aim a bit high, the ratio will come down a bit when you add nodes/OSDs to your cluster to gain capacity. Be sure to balance usage and watch your mon_max_pg_per_osd setting — allowing some headroom for natural variation and for when components fail. > > YMMV. > > — aad > >> On Sep 5, 2020, at 10:34 AM, huxiaoyu(a)horebdata.cn wrote: >> >> Dear Ceph folks, >> >> As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? >> >> thanks a lot, >> >> Samuel >> >> >> >> >> huxiaoyu(a)horebdata.cn >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: PG number per OSD
by Dan van der Ster 06 Sep '20

06 Sep '20

Good question! Did you already observe some performance impact of very large PGs? Which PG locks are you speaking of? Is there perhaps some way to improve this with the op queue shards? (I'm cc'ing Mark in case this is something that the performance team has already looked into). With a 20TB osd, we'll have up to 200GB PGs following the current suggestions -- but even then, backfilling those huge PGs would still be done in under an hour, which seems pretty reasonable IMHO. -- dan On Sat, Sep 5, 2020 at 7:35 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Ceph folks, > > As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? > > thanks a lot, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: PG number per OSD
by Anthony D'Atri 06 Sep '20

06 Sep '20

One factor is RAM usage, that was IIRC the motivation for the lowering of the recommendation of the ratio from 200 to 100. Memory needs also increase during recovery and backfill. When calculating, be sure to consider repllicas. ratio = (pgp_num x replication) / num_osds As HDDs grow the interface though isn’t becoming faster (with SATA at least), and there are only so many IOPS and MB/s that you’re going to get out of one no matter how you slice it. Everything always depends on your use-case and workload, but I suspect that often the bottleneck is the drive, not PG or OSD serialization. For example, do you prize IOPS more, latency, or MB/s? If you don’t care about latency, then you can drive your HDDs harder and get more MB/s throughput out of them, though your average latency might climb to 100ms. Which eg. RBD VM clients probably wouldn’t be too happy about, but which an object service *might* tolerate. Basically in the absence of more info, I would personally suggest aiming at the 150-200 average range, with pgp_num a power of 2. If you aim a bit high, the ratio will come down a bit when you add nodes/OSDs to your cluster to gain capacity. Be sure to balance usage and watch your mon_max_pg_per_osd setting — allowing some headroom for natural variation and for when components fail. YMMV. — aad > On Sep 5, 2020, at 10:34 AM, huxiaoyu(a)horebdata.cn wrote: > > Dear Ceph folks, > > As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? > > thanks a lot, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

librados: rados_cache_pin returning Invalid argument. need help
by Daniel Mezentsev 06 Sep '20

06 Sep '20

Hi ceph-users, Im bit stuck with librados, partcularly rados_cache_pin finction. For some reason it returns "Invalid argument", error code 22. Can't find what im doing wrong. in rados_cache_pin im using the same ioctx and obj name like i use with rados_write - which works just fine. Any help is more then welcome. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

1 0

cephadm orch thinks hosts are offline
by xahare＠gmail.com 06 Sep '20

06 Sep '20

We have a 5 node cluster, all monitors, installed with cephadm. recently, the hosts needed to be rebooted for upgrades, but as we rebooted them, hosts fail their cephadm check. as you can see ceph1 is in quorum and is the host the command is run from. following is the output of ceph -s and ceph orch host ls. ceph orch pause and resume only removed the "offline" status of cephmon-temp, which really is offline. how do we fix ceph orchs confusion? the third is a temp node we had that ceph orch remove host couldnt get rid of. ceph1:~# ceph -s health: HEALTH_WARN 3 hosts fail cephadm check services: mon: 5 daemons, quorum ceph5,ceph4,ceph3,ceph2,ceph1 (age 2d) mgr: ceph3.dmpmih(active, since 3w), standbys: ceph5.pwseyi osd: 30 osds: 30 up (since 2d), 30 in (since 3w) data: pools: 2 pools, 129 pgs objects: 2.29M objects, 8.7 TiB usage: 22 TiB used, 87 TiB / 109 TiB avail pgs: 129 active+clean io: client: 149 KiB/s wr, 0 op/s rd, 14 op/s wr ceph1:~# ceph orch host ls HOST ADDR LABELS STATUS ceph1 ceph1 mon Offline ceph2 ceph2 mon Offline ceph3 ceph3 mon ceph4 ceph4 mon ceph5 ceph5 mon cephmon-temp cephmon-temp Offline

1 0

Re: bug of the year (with compressed omap and lz 1.7(?))
by Dan van der Ster 05 Sep '20

05 Sep '20

Hi Marc, As far as I know, the osdmap corruption occurs with this osd config: bluestore_compression_mode=aggressive bluestore_compression_algorithm=lz4 (My understanding is that if you don't have the above settings, but use pool-specific compression settings instead, then the osdmaps are not compressed, so they would not be corruptable). I prefer to stay on the safe side, though: since we know that lz4 < 1.8.2 can corrupt data if comes from fragmented memory, and we don't know all of the cases where that might be the case, I suggest to use another algorithm or turn off compression until you have upgraded to v14.2.10 or newer. And btw, there was another bug earlier in nautilus where those pool-specific compression settings weren't applied correctly anyway, so I'm not sure they even work yet in 14.2.9. -- dan On Sat, Sep 5, 2020 at 6:12 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > I am still running 14.2.9 with lz4-1.7.5-3. Will I run into this bug > enabling compression on a pool with: > > ceph osd pool set POOL_NAME compression_algorithm COMPRESSION_ALGORITHM > ceph osd pool set POOL_NAME compression_mode COMPRESSION_MODE > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

PG number per OSD
by huxiaoyu＠horebdata.cn 05 Sep '20

05 Sep '20

Dear Ceph folks, As the capacity of one HDD (OSD) is growing bigger and bigger, e.g. from 6TB up to 18TB or even more, should the number of PG per OSD increase as well, e.g. for 200 to 800. As far as i know, the capacity of each PG should be set smaller for performance reasons due to the existence of PG locks, thus shall i set the number of PGs per OSD to 1000 or even 2000? what is the actual reason for not setting the number of PGs per OSD? Is there any practical limations on the number of PGs? thanks a lot, Samuel huxiaoyu(a)horebdata.cn

1 0

bug of the year (with compressed omap and lz 1.7(?))
by Marc Roos 05 Sep '20

05 Sep '20

I am still running 14.2.9 with lz4-1.7.5-3. Will I run into this bug enabling compression on a pool with: ceph osd pool set POOL_NAME compression_algorithm COMPRESSION_ALGORITHM ceph osd pool set POOL_NAME compression_mode COMPRESSION_MODE

1 0

Assignment Help, Essay Help and Dissertation Help
by australianassignmenthelp1＠gmail.com 05 Sep '20

05 Sep '20

Students are often faced with the issue of preparing their assignment during their studies. They often seek for help from professional service providers and one such service provider is Student Life Saviour that offers assistance at: https://studentlifesaviour.com/sg

1 0

Migrating Luminous → Nautilus "Required devices (data, and journal) not present for filestore"
by Stuart Longland 05 Sep '20

05 Sep '20

Hi all, I'm trying to migrate my 5-node cluster to Nautilus… Debian Stretch (I intend to update this once I'm onto Nautilus) as the OS, all 5 machines are combined OSD/MON/MDS nodes. Nodes were originally deployed with an earlier version of Ceph (can't recall now) using filestore on btrfs. Later, after an abortive attempt to move to Bluestore (it ran… veeeeerrryyy ssslllooowwwlllyyy), I moved back to filestore on xfs, which whilst no speed demon, worked well enough. Journal is on the OSD due to space constraints elsewhere in the nodes. So step 1: > If you are unsure whether or not your Luminous cluster has completed a full scrub of all PGs, you can check your cluster’s state by running: > > # ceph osd dump | grep ^flags > > In order to be able to proceed to Nautilus, your OSD map must include the recovery_deletes and purged_snapdirs flags. Tick to that, I see both flags. Step 2. > Make sure your cluster is stable and healthy (no down or recovering OSDs). (Optional, but recommended.) Tick, all healthy. Step 3. > Set the noout flag for the duration of the upgrade. (Optional, but recommended.): Done Step 4. > Upgrade monitors by installing the new packages and restarting the monitor daemons. Okay, so I update /etc/apt/sources.list.d/ceph.list to point to the `nautilus` repository, `apt-get update`, `apt-get dist-upgrade -y`. This goes without a hitch, I now have Ceph 12 binaries on my nodes. > systemctl restart ceph-mon.target Ran that, on all of them (and yes, they are all in the quorum)… > # ceph mon dump | grep min_mon_release > > should report: > > min_mon_release 14 (nautilus) I get this:> root@helium:~# ceph mon dump | grep min_mon_release > dumped monmap epoch 4 No `min_mon_release` mentioned anywhere. I tried re-starting the MON daemons on all 5 nodes, even doing it in parallel using Ansible… no dice. Nothing says what to do at this point. It's not reporting an earlier release, it's just not reporting full stop. Figuring that, well, I *did* update all the monitors, and re-start them, I press on. Step 5. > Upgrade ceph-mgr daemons by installing the new packages and restarting all manager daemons. Done, no issues. > Verify the ceph-mgr daemons are running by checking ceph -s: I get: > root@helium:~# ceph -s > cluster: > id: 45b532b7-aa3d-4754-9906-d4a70b57630c > health: HEALTH_WARN > noout flag(s) set > > services: > mon: 5 daemons, quorum hydrogen,helium,carbon,nitrogen,boron > mgr: hydrogen(active), standbys: carbon, helium > mds: cephfs-1/1/1 up {0=helium=up:active}, 2 up:standby > osd: 5 osds: 5 up, 5 in > flags noout > > data: > pools: 4 pools, 324 pgs > objects: 302.75k objects, 1.13TiB > usage: 3.52TiB used, 5.53TiB / 9.05TiB avail > pgs: 323 active+clean > 1 active+clean+scrubbing+deep > > io: > client: 45.0KiB/s rd, 1.10MiB/s wr, 1op/s rd, 38op/s wr So far so good. Step 6. > Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts: > systemctl restart ceph-osd.target I actually missed this step at first and went on to step 7, but doubled back and did it. Step 7. > If there are any OSDs in the cluster deployed with ceph-disk (e.g., almost any OSDs that were created before the Mimic release), you need to tell ceph-volume to adopt responsibility for starting the daemons. Okay, seems simple enough. Now I did this by mistake earlier, so I'll force it to ensure everything is up to scratch. > root@hydrogen:~# ceph-volume simple scan --force > stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device > stderr: Unknown device, --name=, --path=, or absolute path in /dev/ or /sys expected. > Running command: /sbin/cryptsetup status /dev/sdc1 > --> OSD 5 got scanned and metadata persisted to file: /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json > --> To take over managment of this scanned OSD, and disable ceph-disk and udev, run: > --> ceph-volume simple activate 5 2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe > root@hydrogen:~# ceph-volume simple activate --all > --> activating OSD specified in /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json > --> Required devices (data, and journal) not present for filestore > --> filestore devices found: [u'data'] > --> RuntimeError: Unable to activate filestore OSD due to missing devices Uhh ohh? So, my journal is on the OSD itself: > root@hydrogen:~# ls -l /var/lib/ceph/osd/ceph-5/journal > -rw-r--r-- 1 ceph ceph 21474836480 Aug 29 14:16 /var/lib/ceph/osd/ceph-5/journal > root@hydrogen:~# cat /etc/ceph/osd/5-2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe.json > { > "active": "ok", > "ceph_fsid": "45b532b7-aa3d-4754-9906-d4a70b57630c", > "cluster_name": "ceph", > "data": { > "path": "/dev/sdc1", > "uuid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe" > }, > "fsid": "2a1e6a2b-7742-4cd9-9a39-e4a35ebe74fe", > "keyring": "…censored…", > "magic": "ceph osd volume v026", > "ready": "ready", > "require_osd_release": "", > "systemd": "", > "type": "filestore", > "whoami": 5 > } How do I tell it that the journal is on `/dev/sdc1` as a file within the XFS file store? -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.

1 2

RadosGW and DNS Round-Robin
by DHilsbos＠performair.com 04 Sep '20

04 Sep '20

All; We've been running RadosGW on our nautilus cluster for a while, and we're going to be adding iSCSI capabilities to our cluster, via 2 additional servers. I intend to also run RadosGW on these servers. That begs the question of how to "load balance" these servers. I don't believe that we need true load balancing (i.e. through a dedicated proxy), and I'd rather not add the complexity and single point of failure. The question then is: Does RadosGW play nice with round-robin DNS? The real question here is whether RadosGW maintains internal client state locally between connections. I would expect it's safe, given that it is HTTP, but I'd prefer to verify. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 2

Ceph iSCSI Questions
by DHilsbos＠performair.com 04 Sep '20

04 Sep '20

All; We've used iSCSI to support virtualization for a while, and have used multi-pathing almost the entire time. Now, I'm looking to move from our single box iSCSI hosts to iSCSI on Ceph. We have 2 independent, non-routed, subnets assigned to iSCSI (let's call them 192.168.250.0/24 and 192.168.251.0/24). These subnets are hosted in VLANs 250 and 251, respectively, on our switches. Currently; each target and each initiator have a dedicated network port for each subnet (i.e. 2 NIC per target & 2 NIC per initiator). I have 2 server prepared to setup as Ceph iSCSI targets (let's call them ceph-iscsi1 & cpeh-iscsi2), and I'm wondering about their network configurations. My initial plan is to configure one on the 250 network, and the other on the 251 network. Would it be possible to have both servers on both networks? In other words, can I give ceph-iscsi1 both 192.168.250.200 and 192.168.251.200, and ceph-iscsi2 192.168.250.201 and 192.168.251.201? If that works, I would expect the initiators to see 4 paths to each portal, correct? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 1

Re: cephadm & iSCSI
by Robert Sander 04 Sep '20

04 Sep '20

Hi, I am using 15.2.4 as .5 has not been released. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

cephadm & iSCSI
by Robert Sander 04 Sep '20

04 Sep '20

Hi, yes, I have read https://docs.ceph.com/docs/octopus/cephadm/stability/ and know that the iSCSI support is still under development. But why is the command "ceph orch apply iscsi" available when cephadmin is the orchestrator backend? I am at the stage that I successfully rolled out an iSCSI gateway which is visible in the Ceph dashboard. But when trying to setup a target an error occurs: Sep 04 15:15:58 ceph02 rbd-target-api[124040]: Unable to create the Target definition - Could not load module: iscsi_target_mod Sep 04 15:15:58 ceph02 rbd-target-api[124040]: Unhandled Exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/rtslib_fb/node.py", line 71, in _create_in_cfs_ine os.mkdir(self.path) FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/config/target/iscsi' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/rtslib_fb/fabric.py", line 156, in _check_self self._create_in_cfs_ine('any') File "/usr/lib/python3.6/site-packages/rtslib_fb/node.py", line 74, in _create_in_cfs_ine % self.__class__.__name__) rtslib_fb.utils.RTSLibError: Could not create ISCSIFabricModule in configFS During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 432, in modprobe kmod.Kmod().modprobe(module) File "kmod/kmod.pyx", line 106, in kmod.kmod.Kmod.modprobe File "kmod/kmod.pyx", line 82, in lookup kmod.error.KmodError: Could not modprobe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/bin/rbd-target-api", line 106, in decorated return f(*args, **kwargs) File "/usr/bin/rbd-target-api", line 304, in target target.manage('init') File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/target.py", line 710, in manage 'mutual_password_encryption_enabled']) File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/discovery.py", line 14, in set_discovery_auth_lio iscsi_fabric.clear_discovery_auth_settings() File "/usr/lib/python3.6/site-packages/rtslib_fb/fabric.py", line 224, in clear_discovery_auth_settings self._check_self() File "/usr/lib/python3.6/site-packages/rtslib_fb/fabric.py", line 158, in _check_self modprobe(self.kernel_module) File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 434, in modprobe raise RTSLibError("Could not load module: %s" % module) rtslib_fb.utils.RTSLibError: Could not load module: iscsi_target_mod Solution: "modprobe iscsi_target_mod" on the host itself, the container is not allowed to do that. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 2

how to reduce osd down interval on laggy disk ?
by Alexandre DERUMIER 04 Sep '20

04 Sep '20

Hi, I have sometime laggy ssd drives (intel s3610), maybe a firmware bug or controller bug. (I have tested with differents firmware, don't help). When this occur, I have this kind of error Aug 29 15:43:37 ceph5-7 kernel: [447163.801090] sd 0:0:3:0: Power-on or device reset occurred Just before this, the disk is lagging for 1-2min, and iowait increase a lot. Then monitor is mark it down. (at almost the same time than the disk reset, not sure it's related) Is it possible to reduce the threshold to force the osd down faster ? (maybe 30s for example). (I'm running ceph nautilus 14.2.6) The osd log is: 020-08-29 15:43:37.610 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5618a5f27200, latency = 93.062018 2020-08-29 15:43:37.610 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5618a2956c00, latency = 93.031658 2020-08-29 15:43:37.610 7fa1025b7700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x10543123000~1000 (direct) since 447078s, timeout is 5s 2020-08-29 15:43:37.610 7fa1025b7700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 93.0167s, num_ios = 3072 2020-08-29 15:43:37.642 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56197c0f1200, latency = 92.892911 2020-08-29 15:43:37.642 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56194151db00, latency = 82.923060 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56197c0f0900, latency = 92.611975 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56190ea76000, latency = 92.439439 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x561964172000, latency = 92.861119 2020-08-29 15:43:37.646 7fa0fedb0700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0xf10fa5c000~1000 (direct) since 447091s, timeout is 5s 2020-08-29 15:43:37.646 7fa0fd5ad700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0xe08a1000~1000 (direct) since 447079s, timeout is 5s 2020-08-29 15:43:37.646 7fa0fedb0700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 79.5633s, num_ios = 2048 2020-08-29 15:43:37.646 7fa0fd5ad700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 92.3464s, num_ios = 18 2020-08-29 15:43:37.646 7fa1045bb700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x103a3da1000~1000 (direct) since 447078s, timeout is 5s 2020-08-29 15:43:37.646 7fa0ffdb2700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x100a4d26000~1000 (direct) since 447078s, timeout is 5s 2020-08-29 15:43:37.646 7fa0ffdb2700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 93.022s, num_ios = 1024 2020-08-29 15:43:37.646 7fa1045bb700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 92.5954s, num_ios = 1536 2020-08-29 15:43:37.646 7fa1005b3700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x1003a505000~1000 (direct) since 447079s, timeout is 5s 2020-08-29 15:43:37.646 7fa101db6700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x1012ea45000~1000 (direct) since 447080s, timeout is 5s 2020-08-29 15:43:37.646 7fa1005b3700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 92.3891s, num_ios = 3584 2020-08-29 15:43:37.646 7fa101db6700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 91.1452s, num_ios = 2048 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56191dbce300, latency = 82.924723 2020-08-29 15:43:37.646 7fa0fcdac700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0xfe7a8b7000~1000 (direct) since 447083s, timeout is 5s 2020-08-29 15:43:37.646 7fa0fcdac700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 87.9135s, num_ios = 512 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56196a946300, latency = 82.923270 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5618c70d8f00, latency = 92.344641 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5619748d1500, latency = 92.333448 d for _txc_committed_kv, latency = 91.2568s, txc = 0x56196ab26f00 2020-08-29 15:43:37.774 7fa10e5cf700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.43 down, but it is still running ..... ..... 2020-08-29 15:43:37.774 7fa10e5cf700 0 log_channel(cluster) log [DBG] : map e604017 wrongly marked me down at e604016 2020-08-29 15:43:39.494 7fa0fd5ad700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) aio_submit retries 15 monitor logs --------------- mon1 ---- 2020-08-29 15:43:09.836 7f3ed3030700 0 log_channel(audit) log [DBG] : from='client.835295911 X.X.0.43:0/3364930077' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-08-29 15:43:10.140 7f3ed5835700 0 log_channel(cluster) log [WRN] : Health check update: 715 slow ops, oldest one blocked for 62 sec, daemons [osd.13,osd.14,osd.15,osd.16,osd.17,osd.20,osd.22,osd.23,osd.26,osd.27]... have slow ops. (SLOW_OPS) mon2 ---- 2020-08-29 15:43:20.140 7f3ed5835700 0 log_channel(cluster) log [WRN] : Health check update: 817 slow ops, oldest one blocked for 73 sec, daemons [osd.13,osd.14,osd.15,osd.16,osd.17,osd.20,osd.22,osd.23,osd.26,osd.27]... have slow ops. (SLOW_OPS) mon3 ------ 2020-08-29 15:43:08.066 7f3d213c5700 0 mon.ceph5-2@1(peon) e5 handle_command mon_command({"prefix": "osd blacklist", "blacklistop": "add", "addr": "10.5.0.93:0/955438826"} v 0) v1 2020-08-29 15:43:08.066 7f3d213c5700 0 log_channel(audit) log [INF] : from='client.835295776 X.X.0.92:0/2918303586' entity='client.admin' cmd=[{"prefix": "osd blacklist", "blacklistop": "add", "addr": "10.5.0.93:0/955438826"}]: dispatch 2020-08-29 15:43:09.974 7f3d213c5700 0 mon.ceph5-2@1(peon) e5 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1 2020-08-29 15:43:09.974 7f3d213c5700 0 log_channel(audit) log [DBG] : from='client.? X.X.0.44:0/728500154' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch 2020-08-29 15:43:10.330 7f3d213c5700 0 mon.ceph5-2@1(peon) e5 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1 2020-08-29 15:43

1 0

Multipart upload issue from Java SDK clients
by shubjero 04 Sep '20

04 Sep '20

Good day, I am having an issue with some multipart uploads to radosgw. I recently upgraded my cluster from Mimic to Nautilus and began having problems with multipart uploads from clients using the Java AWS SDK (specifically 1.11.219). I do NOT have issues with multipart uploads with other clients such as the aws2 cli. The error that surfaces is http error 500. Small files that are not multipart uploads work fine with the Java client. I enabled debugging for debug_ms = 1 and debug_rgw = 20 and then performed a multipart upload with the offending client and this is what I see (full log @ https://object.cancercollaboratory.org:9080/swift/v1/ceph-debug-pub/filtere… ) - just a snippet - 2020-09-02 18:03:43.339 7f9277474700 1 -- 172.25.13.93:0/458188962 --> [v2:172.25.12.152:6936/12180,v1:172.25.12.152:6937/12180] -- osd_op(unknown.0.0:6331 17.325 17:a4c2f681:::2020-09-02-18-default.262831163.2-score-rdpc-collab-qa:head [append 0~817] snapc 0=[] ondisk+write+known_if_redirected e1493993) v8 -- 0x55e9fefb9000 con 0x55e9f3e61f80 2020-09-02 18:03:43.339 7f9277474700 2 req 1714 70.432s s3:put_obj op status=-35 2020-09-02 18:03:43.339 7f9277474700 2 req 1714 70.432s s3:put_obj http status=500 haproxy log: Sep 2 13:33:36 localhost haproxy[14495]: 206.108.127.16:46088 [02/Sep/2020:13:32:06.263] radosgw_public_https_11~ radosgw_public_https_11/storage2-r5 0/0/0/71677/89948 500 477 - - ---- 12/12/11/1/0 0/0 {} "PUT /score-rdpc-collab-qa/data/1c9e4395-677c-55c1-8437-b6467c806173?uploadId=2~zTiBm6p4sB2SyMu2cJ_dwd3ECMpELh3&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200902T173202Z&X-Amz-SignedHeaders=host&X-Amz-Expires=518400&X-Amz-Credential=TO1DRTS4H14HJ388I26R%2F20200902%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=225b6f111cc342c97ad48f8a39f34af044647ac04da532ef949d770509e65f76 HTTP/1.1" Does anyone know if there may be incompatiblities with Java AWS SDK v1? I see that v2 is out so one of our devs is going to run some tests using Java AWS SDK v2 but this may take a bit of time. Thanks all, Jared Baker Ontario Institute for Cancer Research

1 1

Change Roadrunner Email Password | RR Email Help
by elouisesng5445＠gmail.com 04 Sep '20

04 Sep '20

Roadrunner Recovery by Roadrunner Email Password is a easy process. We have often forgotten our new password because we don't want it, or it's always stolen because of account. So the network is designed to overcome these Roadrunner email authentication failures, where we can find a path to our solution. When you Change Roadrunner Password you don't have to worry at all, because resetting it is very fast. You will find the full description of the Change Roadrunner password here, in this article. Also we'll send you some tips and tricks to keep your Roadrunner email account safe. for More Information Please Visit: http://www.emailscustomercare.com/change-roadrunner-password/

1 0

International B2B Marketplace
by Grace Stefan 04 Sep '20

04 Sep '20

Exporthub is an international <a href="https://www.exporthub.com/">b2b marketplace</a> catering to thousands of globally active manufacturers & suppliers from all industries with its safest trade services. Exporthub is dedicated to making international trading more comfortable than ever. Our team of experts is keen to provide all global buyers and suppliers with tools to help them succeed in their particular industry. Our range of services allows suppliers to connect with buyers worldwide for their products. As a B2B trade portal, we provide a B2B platform that lets all American, Indian, African, and Chinese B2B site to enhance their network and connect with businesses across their borders.

1 0

Did I use Cash.app/help to utilize it without telephone number?
by mary smith 04 Sep '20

04 Sep '20

You don't have a telephone number and you attempted to sign in to your Cash app account. Most importantly, click on the profile symbol and afterward select the Personal tab. Presently, you have to include any missing email address or telephone number. It fundamentally needs to get an affirmation code. Hello, click here and get Cash.app/help for any issues. https://www.cashapp-customerservice.com/

1 0

Actual block size of osd
by Ragan, Tj (Dr.) 04 Sep '20

04 Sep '20

Hi All, Does anyone know how to get the actual block size used by an osd? I’m trying to evaluate 4k vs 64k min_alloc_size_hdd and want to verify that the newly created osds are actually using the expected block size. Thanks, -TJ Ragan

2 1

Re: Change fsid of Ceph cluster after splitting it into two clusters
by Dan van der Ster 03 Sep '20

03 Sep '20

Hi Wido, Out of curiosity, did you ever work out how to do this? Cheers, Dan On Tue, Feb 12, 2019 at 6:17 PM Wido den Hollander <wido(a)42on.com> wrote: > > Hi, > > I've got a situation where I need to split a Ceph cluster into two. > > This cluster is currently running a mix of RBD and RGW and in this case > I am splitting it into two different clusters. > > A difficult thing to do, but it's possible. > > One problem that stays though is that after the split both Ceph clusters > have the same fsid and that might be confusing. > > Is there a way to change the fsid of an existing cluster? > > Injecting an updated MONMAP and OSDMAP into the cluster? > > It's no problem if this has to be done offline, but I'm just wondering > if this is possible. > > Wido > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

2 1

Is it possible to change the cluster network on a production ceph?
by psousa＠alticelabs.com 03 Sep '20

03 Sep '20

Hello people, I am trying to change the cluster network in a production ceph. I'm having problems, after changing the ceph.conf file and restarting a osd the cluster is always going to HEALTH_ERROR with blocked requests. Only by returning to the previous configuration and restarting the same osd make the cluster going to OK. So, my question is: is it possible to change the cluster network on a production ceph without stopping the service and destroying data? If so, how? Anticipated thanks Paulo Sousa

2 1