April 2021 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 9

Upgrade tips from Luminous to Nautilus?
by Mark Schouten 08 Jul '21

08 Jul '21

Hi, We've done our fair share of Ceph cluster upgrades since Hammer, and have not seen much problems with them. I'm now at the point that I have to upgrade a rather large cluster running Luminous and I would like to hear from other users if they have experiences with issues I can expect so that I can anticipate on them beforehand. As said, the cluster is running Luminous (12.2.13) and has the following services active: services: mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 mgr: osdnode01(active), standbys: osdnode02, osdnode03 mds: pmrb-3/3/3 up {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 up:standby osd: 116 osds: 116 up, 116 in; rgw: 3 daemons active Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster is 1.01PiB. We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is a total of 5760 pgs. { "rule_id": 1, "rule_name": "hdd-data", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "ssd-data", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } rbd -> crush_rule: hdd-data .rgw.root -> crush_rule: hdd-data default.rgw.control -> crush_rule: hdd-data default.rgw.data.root -> crush_rule: ssd-data default.rgw.gc -> crush_rule: ssd-data default.rgw.log -> crush_rule: ssd-data default.rgw.users.uid -> crush_rule: hdd-data default.rgw.usage -> crush_rule: ssd-data default.rgw.users.email -> crush_rule: hdd-data default.rgw.users.keys -> crush_rule: hdd-data default.rgw.meta -> crush_rule: hdd-data default.rgw.buckets.index -> crush_rule: ssd-data default.rgw.buckets.data -> crush_rule: hdd-data default.rgw.users.swift -> crush_rule: hdd-data default.rgw.buckets.non-ec -> crush_rule: ssd-data DB0475 -> crush_rule: hdd-data cephfs_pmrb_data -> crush_rule: hdd-data cephfs_pmrb_metadata -> crush_rule: ssd-data All but four clients are running Luminous, the four are running Jewel (that needs upgrading before proceeding with this upgrade). So, normally, I would 'just' upgrade all Ceph packages on the monitor-nodes and restart mons and then mgrs. After that, I would upgrade all Ceph packages on the OSD nodes and restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting the OSD's will probably take a while. If anyone has a hint on what I should expect to cause some extra load or waiting time, that would be great. Obviously, we have read https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking for real world experiences. Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

4 11

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Recommendations on problem with PG
by Gabriel Medve 23 May '21

23 May '21

Hi, We have a problem with a PG that was inconsistent, currently the PG in our cluster have 3 copies. It was not possible for us to repair this pg with "ceph pg repair" (This PG is in osd 14,1,2) so we deleted some of the copies of osd 14 with the following command. ceph-objectstore-tool --data-path /var/lib/ceph/osd.14/ --pgid 22.f --op remove --force This caused an automatic attempt to create the missing copy entering the backfilling state, but when doing this it crashed osd 1 and 2 and threw the IOPS to 0, freezing the cluster. Is there any way to remove this entire pg or try to recreate the missing copy or ignore it completely? It causes instability in the cluster. Thank you, I await comments -- Untitled Document ------------------------------------------------------------------------ Gabriel I. Medve

2 2

ceph-ansible in Pacific and beyond?
by Matthew Vernon 20 May '21

20 May '21

Hi, I caught up with Sage's talk on what to expect in Pacific ( https://www.youtube.com/watch?v=PVtn53MbxTc ) and there was no mention of ceph-ansible at all. Is it going to continue to be supported? We use it (and uncontainerised packages) for all our clusters, so I'd be a bit alarmed if it was going to go away... Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

17 25

libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
by Markus Kienast 17 May '21

17 May '21

I am seeing these messages when booting from RBD and booting hangs there. libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks. How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented. This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem. Cheers Markus

4 14

RGW: Multiple Site does not sync olds data
by 特木勒 17 May '21

17 May '21

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

3 17

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

x-amz-request-id logging with beast + rgw (ceph 15.2.10/containerized)?
by David Orman 07 May '21

07 May '21

Hi, Is there any way to log the x-amz-request-id along with the request in the rgw logs? We're using beast and don't see an option in the configuration documentation to add headers to the request lines. We use centralized logging and would like to be able to search all layers of the request path (edge, lbs, ceph, etc) with a x-amz-request-id. Right now, all we see is this: debug 2021-04-01T15:55:31.105+0000 7f54e599b700 1 beast: 0x7f5604c806b0: x.x.x.x - - [2021-04-01T15:55:31.105455+0000] "PUT /path/object HTTP/1.1" 200 556 - "aws-sdk-go/1.36.15 (go1.15.3; linux; amd64)" - We've also tried this: ceph config set global rgw_enable_ops_log true ceph config set global rgw_ops_log_socket_path /tmp/testlog After doing this, inside the rgw container, we can socat - UNIX-CONNECT:/tmp/testlog and see the log entries being recorded that we want, but there has to be a better way to do this, where the logs are emitted like the request logs above by beast, so that we can handle it using journald. If there's an alternative that would accomplish the same thing, we're very open to suggestions. Thank you, David

3 6

Upgrade and lost osds Operation not permitted
by Behzad Khoshbakhti 07 May '21

07 May '21

Hi all, As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the manual upgrade using the precompiled packages, the OSDs was down with the following messages: root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 --path /var/lib/ceph/osd/ceph-2 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/systemctl enable ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 Running command: /usr/bin/systemctl start ceph-osd@2 --> ceph-volume lvm activate successful for osd ID: 2 Content of /var/log/ceph/ceph-osd.2.log 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 (ceph:ceph) 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-osd, pid 5484 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore empty --pid-file 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or directory* root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 â— ceph-osd(a)2.service - Ceph object storage daemon osd.2 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 +0430; 50s ago Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) Main PID: 5484 (code=exited, status=1/FAILURE) Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled restart job, restart counter is at 3. Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon osd.2. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request repeated too quickly. Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with result 'exit-code'. Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage daemon osd.2. root@osd03:/var/lib/ceph/osd/ceph-2# root@osd03:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk loop0 7:0 0 55.5M 1 loop /snap/core18/1988 loop1 7:1 0 69.9M 1 loop /snap/lxd/19188 loop2 7:2 0 55.5M 1 loop /snap/core18/1997 loop3 7:3 0 70.4M 1 loop /snap/lxd/19647 loop4 7:4 0 32.3M 1 loop /snap/snapd/11402 loop5 7:5 0 32.3M 1 loop /snap/snapd/11107 sda 8:0 0 80G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 79G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / sdb 8:16 0 16G 0 disk └─sdb1 8:17 0 16G 0 part └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 b89--bcd4--105b2df26352 253:1 0 16G 0 lvm root@osd03:~# root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) root@osd03:/var/lib/ceph/osd/ceph-2# any help is much appreciated -- Regards Behzad Khoshbakhti Computer Network Engineer (CCIE #58887)

6 14

Re: How to set bluestore_rocksdb_options_annex
by ceph＠elchaka.de 05 May '21

05 May '21

Hello Anthony, it was introduced in octopus 15.2.10 See: https://docs.ceph.com/en/latest/releases/octopus/ Do you know how you would set it in pacific? :) Guess, there shouldnt be much difference... Thank you Mehmet Am 28. April 2021 19:21:19 MESZ schrieb Anthony D'Atri <anthony.datri(a)gmail.com>: >I think that’s new with Pacific. > >> On Apr 28, 2021, at 1:26 AM, ceph(a)elchaka.de wrote: >> >> >> >> Hello, >> >> I have an octopus cluster and want to change some values - but i >cannot find any documentation on how to set values(multiple) with >> >> bluestore_rocksdb_options_annex >> >> Could someone give me some examples. >> I would like to do this like ceph config set ... >> >> Thanks in advice >> Mehmet >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5

Failed cephadm Upgrade - ValueError
by Ashley Merrick 04 May '21

04 May '21

Hello All,I was running 15.2.8 via cephadm on docker Ubuntu 20.04I just attempted to upgrade to 16.2.1 via the automated method, it successfully upgraded the mon/mgr/mds and some OSD's, however it then failed on an OSD and hasn't been able to pass even after stopping and restarting the upgrade.It reported the following ""message": "Error: UPGRADEREDEPLOYDAEMON: Upgrading daemon osd.35 on host sn-s01 failed.""If I run 'ceph health detail' I get lot's of the following error : "ValueError: not enough values to unpack (expected 2, got 1)" throughout the detail reportUpon googling, it looks like I am hitting something along the lines of https://158.69.68.89/issues/48924 & https://tracker.ceph.com/issues/49522What do I need to do to either get around this bug, or a way I can manually upgrade the remaining ceph OSD's to 16.2.1, currently my cluster is working but the last OSD it failed to upgrade is currently offline (I guess as no image attached to it now as it failed to pull it), and I have a cluster with OSD's from not 15.2.8 and 16.2.1Thanks Sent via MXlogin

2 4

OSD id 241 != my id 248: conversion from "ceph-disk" to "ceph-volume simple" destroys OSDs
by Frank Schilder 04 May '21

04 May '21

Hi all, this is a follow-up on "reboot breaks OSDs converted from ceph-disk to ceph-volume simple". I converted a number of ceph-disk OSDs to ceph-volume using "simple scan" and "simple activate". Somewhere along the way, the OSDs meta-data gets rigged and the prominent symptom is that the symlink block is changes from a part-uuid target to an unstable device name target like: before conversion: block -> /dev/disk/by-partuuid/9123be91-7620-495a-a9b7-cc85b1de24b7 after conversion: block -> /dev/sdj2 This is a huge problem as the "after conversion" device names are unstable. I have now a cluster that I cannot reboot servers on due to this problem. OSDs randomly re-assigned devices will refuse to start with: 2021-03-02 15:56:21.709 7fb7c2549b80 -1 OSD id 241 != my id 248 Please help me with getting out of this mess. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 7

Cannot create issue in bugtracker
by Tobias Urdin 03 May '21

03 May '21

Hello, Is it only me that's getting Internal error when trying to create issues in the bugtracker for some day or two? https://tracker.ceph.com/issues/new Best regards

2 2

How radosgw works ?
by Fabrice Bacchella 02 May '21

02 May '21

I'm trying to understand what and where radosgw listen ? There is a lot of contradictory or redundant informations about that. First about the contradictory informations for the socket. At https://docs.ceph.com/en/pacific/radosgw/config-ref/ <https://docs.ceph.com/en/pacific/radosgw/config-ref/>, it says rgw_socket_path, but at https://docs.ceph.com/en/pacific/man/8/radosgw/ <https://docs.ceph.com/en/pacific/man/8/radosgw/> is says 'rgw socket path' That problem is quite common in the ceph documentation. Are both value accepted ? Next about some naming, or binding IP. Where it's defined, and how ? You have: rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 port=443 ..." rgw_host = rgw_port = rgw_dns_name = That's a lot of redundancy, or contradictory informations. What is the purpose of each one ? What is the difference between rgw_frontends = ".. port = ..." and rgw_port = ? Or rgw_host and rgw_dns_name. What is the difference ? The documentation provides no help at all: rgw_dns_name Description: The DNS name of the served domain. See also the hostnames setting within regions. The description says nothing new, it just repeat the field name. Is one of them used by the manager for communication ? I already had the problem for the entry in the certificate used by the frontend, it used an IP coming from nowhere. If a fcgi is used, how the manager find the endpoint ?

2 2

Best distro to run ceph.
by Peter Childs 01 May '21

01 May '21

I'm trying to set up a new ceph cluster, and I've hit a bit of a blank. I started off with centos7 and cephadm. Worked fine to a point, except I had to upgrade podman but it mostly worked with octopus. Since this is a fresh cluster and hence no data at risk, I decided to jump straight into Pacific when it came out and upgrade. Which is where my trouble began. Mostly because Pacific needs a version on lvm later than what's in centos7. I can't upgrade to centos8 as my boot drives are not supported by centos8 due to the way redhst disabled lots of disk drivers. I think I'm looking at Ubuntu or debian. Given cephadm has a very limited set of depends it would be good to have a supported matrix, it would also be good to have a check in cephadm on upgrade, that says no I won't upgrade if the version of lvm2 is too low on any host and let's the admin fix the issue and try again. I was thinking to upgrade to centos8 for this project anyway until I relised that centos8 can't support my hardware I've inherited. But currently I've got a broken cluster unless I can workout some way to upgrade lvm in centos7. Peter.

3 3

Large OSD Performance: osd_op_num_shards, osd_op_num_threads_per_shard
by Dave Hall 30 Apr '21

30 Apr '21

Hello, I noticed a couple unanswered questions on this topic from a while back. It seems, however, worth asking whether adjusting either or both of the subject attributes could improve performance with large HDD OSDs (mine are 12TB SAS). In the previous posts on this topic the writers indicated that they had experimented with increasing either or both of osd_op_num_shards and osd_op_num_threads_per_shard and had seen performance improvement. Like myself, the writers wondering about any limitations or pitfalls relating to such adjustments. Since I would rather not take chances with a 500TB production cluster I am asking for guidance from this list. BTW, my cluster is currently running Nautilus 14.2.6 (stock Debian packages). Thank you. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Re: one of 3 monitors keeps going down
by Eugen Block 30 Apr '21

30 Apr '21

Have you checked for disk failure? dmesg, smartctl etc. ? Zitat von "Robert W. Eckert" <rob(a)rob.eckert.name>: > I worked through that workflow- but it seems like the one monitor > will run for a while - anywhere from an hour to a day, then just stop. > > This machine is running on AMD hardware (3600X CPU on X570 chipset) > while my other two are running on old intel. > > I did find this in the service logs > > 2021-04-30T16:02:40.135+0000 7f5d0a94f700 -1 rocksdb: submit_common > error: Corruption: block checksum mismatch: expected 395334538, got > 4289108204 in /var/lib/ceph/mon/ceph-cube/store.db/073501.sst > offset 36769734 size 84730 code = 2 Rocksdb transaction: > > I am attaching the output of > journalctl -u ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube.service > > The error appears to be here: > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -61> > 2021-04-30T16:02:38.700+0000 7f5d21332700 4 mon.cube(a)-1(???).mgr > e702 active server: > [v2:192.168.2.199:6834/1641928541,v1:192.168.2.199:6835/1641928541](2184157) > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -60> > 2021-04-30T16:02:38.700+0000 7f5d21332700 4 mon.cube(a)-1(???).mgr > e702 mkfs or daemon transitioned to available, loading commands > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -59> > 2021-04-30T16:02:38.701+0000 7f5d21332700 4 set_mon_vals no > callback set > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -58> > 2021-04-30T16:02:38.701+0000 7f5d21332700 10 set_mon_vals > client_cache_size = 32768 > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -57> > 2021-04-30T16:02:38.701+0000 7f5d21332700 10 set_mon_vals > container_image = > docker.io/ceph/ceph@sha256:15b15fb7a708970f1b734285ac08aef45dcd76e86866af37412d041e00853743 > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -56> > 2021-04-30T16:02:38.701+0000 7f5d21332700 10 set_mon_vals > log_to_syslog = true > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -55> > 2021-04-30T16:02:38.701+0000 7f5d21332700 10 set_mon_vals > mon_data_avail_warn = 10 > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -54> > 2021-04-30T16:02:38.701+0000 7f5d21332700 10 set_mon_vals > mon_warn_on_insecure_global_id_reclaim_allowed = true > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -53> > 2021-04-30T16:02:38.701+0000 7f5d21332700 4 set_mon_vals no > callback set > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -52> > 2021-04-30T16:02:38.702+0000 7f5d21332700 2 auth: KeyRing::load: > loaded key file /var/lib/ceph/mon/ceph-cube/keyring > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -51> > 2021-04-30T16:02:38.702+0000 7f5d1095b700 3 rocksdb: > [db_impl/db_impl_compaction_flush.cc:2808] Compaction error: > Corruption: block checksum mismatch: expected 395334538, got > 4289108204 in /var/lib/ceph/mon/ceph- cube/store.db/073501.sst > offset 36769734 size 84730 > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -50> > 2021-04-30T16:02:38.702+0000 7f5d21332700 5 asok(0x56327d226000) > register_command compact hook 0x56327e028700 > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -49> > 2021-04-30T16:02:38.702+0000 7f5d1095b700 4 rocksdb: (Original Log > Time 2021/04/30-16:02:38.703267) [compaction/compaction_job.cc:760] > [default] compacted to: base level 6 level multiplier 10.00 max > bytes base 268435456 files[5 0 0 0 0 0 2] max score 0.00, MB/sec: > 11035.6 rd, 0.0 wr, level 6, files in(5, 2) out(1) MB in(32.1, > 126.7) out(0.0), read-write-amplify(5.0) write-amplify(0.0) > Corruption: block checksum mismatch: expected 395334538, got > 4289108204 in /var/lib/ceph/mon/ceph-cube/store.db/073501.sst > offset 36769734 size 84730, records in: 7670, records dropped: 6759 > output_compres > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -48> > 2021-04-30T16:02:38.702+0000 7f5d1095b700 4 rocksdb: (Original Log > Time 2021/04/30-16:02:38.703283) EVENT_LOG_v1 {"time_micros": > 1619798558703277, "job": 3, "event": "compaction_finished", > "compaction_time_micros": 15085, "compaction_time_cpu_micros": > 11937, "output_level": 6, "num_output_files": 1, > "total_output_size": 12627499, "num_input_records": 7670, > "num_output_records": 911, "num_subcompactions": 1, > "output_compression": "NoCompression", > "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": > 0, "lsm_state": [5, 0, 0, 0, 0, 0, 2]} > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -47> > 2021-04-30T16:02:38.702+0000 7f5d1095b700 2 rocksdb: > [db_impl/db_impl_compaction_flush.cc:2344] Waiting after background > compaction error: Corruption: block checksum mismatch: expected > 395334538, got 4289108204 in > /var/lib/ceph/mon/ceph-cube/store.db/073501.sst offset 36769734 > size 84730, Accumulated background error counts: 1 > Apr 30 12:02:40 cube.robeckert.us conmon[41474]: debug -46> > 2021-04-30T16:02:38.702+0000 7f5d21332700 5 asok(0x56327d226000) > register_command smart hook 0x56327e028700 > > > This is running the latest pacific container, but I was seeing the > same issue in octopus. > > The container runs under podman on rhel 8, and the > /var/lib/ceph/mon/ceph-cube is mapped to > /var/lib/ceph/fe3a7cb0-69ca-11eb-8d45-c86000d08867/mon.cube.service > on the nvme boot drive, which has plenty of space. > > To recover I run a script that will stop the monitor on another > host, copy the store.db directory then start up, and it syncs right > up. > > > > Thanks, > Rob > > > > > > -----Original Message----- > From: Sebastian Wagner <sewagner(a)redhat.com> > Sent: Thursday, April 29, 2021 7:44 AM > To: Eugen Block <eblock(a)nde.ag>; ceph-users(a)ceph.io > Subject: [ceph-users] Re: one of 3 monitors keeps going down > > Right, here are the docs for that workflow: > > https://docs.ceph.com/en/latest/cephadm/mon/#mon-service > > Am 29.04.21 um 13:13 schrieb Eugen Block: >> Hi, >> >> instead of copying MON data to this one did you also try to redeploy >> the MON container entirely so it gets a fresh start? >> >> >> Zitat von "Robert W. Eckert" <rob(a)rob.eckert.name>: >> >>> Hi, >>> On a daily basis, one of my monitors goes down >>> >>> [root@cube ~]# ceph health detail >>> HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum >>> rhel1.robeckert.us,story [WRN] CEPHADM_FAILED_DAEMON: 1 failed >>> cephadm daemon(s) >>> daemon mon.cube on cube.robeckert.us is in error state [WRN] >>> MON_DOWN: 1/3 mons down, quorum rhel1.robeckert.us,story >>> mon.cube (rank 2) addr >>> [v2:192.168.2.142:3300/0,v1:192.168.2.142:6789/0] is down (out of >>> quorum) [root@cube ~]# ceph --version ceph version 15.2.11 >>> (e3523634d9c2227df9af89a4eac33d16738c49cb) >>> octopus (stable) >>> >>> I have a script that will copy the mon data from another server and >>> it restarts and runs well for a while. >>> >>> It is always the same monitor, and when I look at the logs the only >>> thing I really see is the cephadm log showing it down >>> >>> 2021-04-28 10:07:26,173 DEBUG Running command: /usr/bin/podman >>> --version >>> 2021-04-28 10:07:26,217 DEBUG /usr/bin/podman: stdout podman version >>> 2.2.1 >>> 2021-04-28 10:07:26,222 DEBUG Running command: /usr/bin/podman >>> inspect --format >>> {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index >>> .Config.Labels "io.ceph.version"}} >>> ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867-osd.2 >>> 2021-04-28 10:07:26,326 DEBUG /usr/bin/podman: stdout >>> fab17e5242eb4875e266df19ca89b596a2f2b1d470273a99ff71da2ae81eeb3c,dock >>> er.io/ceph/ceph:v15,5b724076c58f97872fc2f7701e8405ec809047d71528f79da >>> 452188daf2af72e,2021-04-26 >>> 17:13:15.54183375 -0400 EDT, >>> 2021-04-28 10:07:26,328 DEBUG Running command: systemctl is-enabled >>> ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube<mailto:ceph-fe3a7c >>> b0-69ca-11eb-8d45-c86000d08867(a)mon.cube> >>> >>> 2021-04-28 10:07:26,334 DEBUG systemctl: stdout enabled >>> 2021-04-28 10:07:26,335 DEBUG Running command: systemctl is-active >>> ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube<mailto:ceph-fe3a7c >>> b0-69ca-11eb-8d45-c86000d08867(a)mon.cube> >>> >>> 2021-04-28 10:07:26,340 DEBUG systemctl: stdout failed >>> 2021-04-28 10:07:26,340 DEBUG Running command: /usr/bin/podman >>> --version >>> 2021-04-28 10:07:26,395 DEBUG /usr/bin/podman: stdout podman version >>> 2.2.1 >>> 2021-04-28 10:07:26,402 DEBUG Running command: /usr/bin/podman >>> inspect --format >>> {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index >>> .Config.Labels "io.ceph.version"}} >>> ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867-mon.cube >>> 2021-04-28 10:07:26,526 DEBUG /usr/bin/podman: stdout >>> 04e7c673cbacf5160427b0c3eb2f0948b2f15d02c58bd1d9dd14f975a84cfc6f,dock >>> er.io/ceph/ceph:v15,5b724076c58f97872fc2f7701e8405ec809047d71528f79da >>> 452188daf2af72e,2021-04-28 >>> 08:54:57.614847512 -0400 EDT, >>> >>> I don't know if it matters, but this server is an AMD 3600XT while >>> my other two servers which have had no issues are intel based. >>> >>> The root file system was originally on a SSD, and I switched to NVME, >>> so I eliminated controller or drive issues. (I didn't see anything >>> in dmesg anyway) >>> >>> If someone could point me in the right direction on where to >>> troubleshoot next, I would appreciate it. >>> >>> Thanks, >>> Rob Eckert >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >>> email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

2 1

Performance questions - 4 node (commodity) cluster - what to expect (and what not ;-)
by Schmid, Michael 30 Apr '21

30 Apr '21

Hello folks, I am new to ceph and at the moment I am doing some performance tests with a 4 node ceph-cluster (pacific, 16.2.1). Node hardware (4 identical nodes): * DELL 3620 workstation * Intel Quad-Core i7-6700(a)3.4 GHz * 8 GB RAM * Debian Buster (base system, installed a dedicated on Patriot Burst 120 GB SATA-SSD) * HP 530SPF+ 10 GBit dual-port NIC (tested with iperf to 9.4 GBit/s from node to node) * 1 x Kingston KC2500 M2 NVMe PCIe SSD (500 GB, NO power loss protection !) * 3 x Seagate Barracuda SATA disk drives (7200 rpm, 500 GB) After bootstrapping a containerized (docker) ceph-cluster, I did some performance tests on the NVMe storage by creating a storage pool called „ssdpool“, consisting of 4 OSDs per (one) NVMe device (per node). A first write-performance test yields ============= root@ceph1:~# rados bench -p ssdpool 10 write -b 4M -t 16 --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ceph1_78 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 30 14 55.997 56 0.0209977 0.493427 2 16 53 37 73.9903 92 0.0264305 0.692179 3 16 76 60 79.9871 92 0.559505 0.664204 4 16 99 83 82.9879 92 0.609332 0.721016 5 16 116 100 79.9889 68 0.686093 0.698084 6 16 132 116 77.3224 64 1.19715 0.731808 7 16 153 137 78.2741 84 0.622646 0.755812 8 16 171 155 77.486 72 0.25409 0.764022 9 16 192 176 78.2076 84 0.968321 0.775292 10 16 214 198 79.1856 88 0.401339 0.766764 11 1 214 213 77.4408 60 0.969693 0.784002 Total time run: 11.0698 Total writes made: 214 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 77.3272 Stddev Bandwidth: 13.7722 Max bandwidth (MB/sec): 92 Min bandwidth (MB/sec): 56 Average IOPS: 19 Stddev IOPS: 3.44304 Max IOPS: 23 Min IOPS: 14 Average Latency(s): 0.785372 Stddev Latency(s): 0.49011 Max latency(s): 2.16532 Min latency(s): 0.0144995 ============= ... and I think that 80 MB/s throughput is a very poor result in conjunction with NVMe devices and 10 GBit nics. A bare write-test (with fsync=0 option) of the NVMe drives yields a write throughput of round about 800 MB/s per device ... the second test (with fsync=1) drops performance to 200 MB/s. ============= root@ceph1:/home/mschmid# fio --rw=randwrite --name=IOPS-write --bs=1024k --direct=1 --filename=/dev/nvme0n1 --numjobs=4 --ioengine=libaio --iodepth=32 --refill_buffers --group_reporting --runtime=30 --time_based --fsync=0 IOPS-write: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32... fio-3.12 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=723MiB/s][w=722 IOPS][eta 00m:00s] IOPS-write: (groupid=0, jobs=4): err= 0: pid=31585: Thu Apr 29 15:15:03 2021 write: IOPS=740, BW=740MiB/s (776MB/s)(21.8GiB/30206msec); 0 zone resets slat (usec): min=16, max=810, avg=106.48, stdev=30.48 clat (msec): min=7, max=1110, avg=172.09, stdev=120.18 lat (msec): min=7, max=1110, avg=172.19, stdev=120.18 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 63], | 30.00th=[ 115], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 220], 90.00th=[ 264], 95.00th=[ 368], | 99.00th=[ 667], 99.50th=[ 751], 99.90th=[ 894], 99.95th=[ 986], | 99.99th=[ 1036] bw ( KiB/s): min=22528, max=639744, per=25.02%, avg=189649.94, stdev=113845.69, samples=240 iops : min= 22, max= 624, avg=185.11, stdev=111.18, samples=240 lat (msec) : 10=0.01%, 20=0.19%, 50=6.43%, 100=20.29%, 250=61.52% lat (msec) : 500=8.21%, 750=2.85%, 1000=0.47% cpu : usr=11.87%, sys=2.05%, ctx=13141, majf=0, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.4%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,22359,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=740MiB/s (776MB/s), 740MiB/s-740MiB/s (776MB/s-776MB/s), io=21.8GiB (23.4GB), run=30206-30206msec Disk stats (read/write): nvme0n1: ios=0/89150, merge=0/0, ticks=0/15065724, in_queue=15118720, util=99.75% ============= Furthermore an IOPS-test on the NVMe device with block-size 4k shows round about 1000 IOPS with fsnyc=1 and 35000 IOPS with fsync=0. To my question: As CPU- and network-load seem to be low during my tests, I would like to know, which bottleneck can cause such a huge performance drop between the bare hardware-performance of the nvme-drives and the write-speeds in the rados benchmark. Could the missing power loss protection (fsync=1) be the problem, or what throughput should one expect to be normal in such a setup? Thanks for every advice! Best regards, Michael

4 4

Re: Specify monitor IP when CIDR detection fails
by Stephen Smith6 30 Apr '21

30 Apr '21

1 0

Specify monitor IP when CIDR detection fails
by Stephen Smith6 30 Apr '21

30 Apr '21

1 0

cephadm upgrade from v15.11 to pacific fails all the times
by Ackermann, Christoph 30 Apr '21

30 Apr '21

Dear gents, to get handy with cephadm upgrade path and in general (we heavily use old style "ceph-deploy" Octopus based production clusters), we decided to do some tests with a vanilla cluster running 15.2.11 based on Centos8 on top of vSphere. Deployment of Octopus cluster runs very well and we are excited about this new technique and all the possibilities. No errors no clues... :-) Unfortunately upgrade fails to Pacific (16.2.0 or 16.2.1) either original docker or quay.ceph.io/ceph-ci/ceph:pacific images all the time. We use a small setup (3 mons, 2 mgrs, some osds) This is the upgrade behaviour: Upgrade of both MGR's seems to be ok but we get this: 2021-04-29T15:35:19.903111+0200 mgr.c0n00.vnxaqu [DBG] daemon mgr.c0n00.vnxaqu container digest correct 2021-04-29T15:35:19.903206+0200 mgr.c0n00.vnxaqu [DBG] daemon mgr.c0n00.vnxaqu deployed by correct version 2021-04-29T15:35:19.903298+0200 mgr.c0n00.vnxaqu [DBG] daemon mgr.c0n01.gstlmw container digest correct 2021-04-29T15:35:19.903378+0200 mgr.c0n00.vnxaqu [DBG] daemon mgr.c0n01.gstlmw *not deployed by correct version* After this the upgrade process stucks completely. Although you have a running cluster (minus one monitor daemon): [root@c0n00 ~]# ceph -s cluster: id: 5541c866-a8fe-11eb-b604-005056b8f1bf health: HEALTH_WARN * 3 hosts fail cephadm check* services: mon: 2 daemons, quorum c0n00,c0n02 (age 68m) mgr: c0n00.bmtvpr(active, since 68m), standbys: c0n01.jwfuca osd: 4 osds: 4 up (since 63m), 4 in (since 62m) [..] progress: Upgrade to 16.2.1-257-g717ce59b (0s) [=...........................] { "target_image": " quay.ceph.io/ceph-ci/ceph@sha256:d0f624287378fe63fc4c30bccc9f82bfe0e42e62381c0a3d0d3d86d985f5d788", "in_progress": true, "services_complete": [ "mgr" ], "progress": "2/19 ceph daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception" [root@c0n00 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID alertmanager.c0n00 c0n00 running (56m) 4m ago 16h 0.20.0 0881eb8f169f 30d9eff06ce2 crash.c0n00 c0n00 running (56m) 4m ago 16h 15.2.11 9d01da634b8f 91d3e4d0e14d crash.c0n01 c0n01 host is offline 16h ago 16h 15.2.11 9d01da634b8f 0ff4a20021df crash.c0n02 c0n02 host is offline 16h ago 16h 15.2.11 9d01da634b8f 0253e6bb29a0 crash.c0n03 c0n03 host is offline 16h ago 16h 15.2.11 9d01da634b8f 291ce4f8b854 grafana.c0n00 c0n00 running (56m) 4m ago 16h 6.7.4 80728b29ad3f 46d77b695da5 mgr.c0n00.bmtvpr c0n00 *:8443,9283 running (56m) 4m ago 16h 16.2.1-257-g717ce59b 3be927f015dd 94a7008ccb4f mgr.c0n01.jwfuca c0n01 host is offline 16h ago 16h 16.2.1-257-g717ce59b 3be927f015dd 766ada65efa9 mon.c0n00 c0n00 running (56m) 4m ago 16h 15.2.11 9d01da634b8f b9f270cd99e2 mon.c0n02 c0n02 host is offline 16h ago 16h 15.2.11 9d01da634b8f a90c21bfd49e node-exporter.c0n00 c0n00 running (56m) 4m ago 16h 0.18.1 e5a616e4b9cf eb1306811c6c node-exporter.c0n01 c0n01 host is offline 16h ago 16h 0.18.1 e5a616e4b9cf 093a72542d3e node-exporter.c0n02 c0n02 host is offline 16h ago 16h 0.18.1 e5a616e4b9cf 785531f5d6cf node-exporter.c0n03 c0n03 host is offline 16h ago 16h 0.18.1 e5a616e4b9cf 074fac77e17c osd.0 c0n02 host is offline 16h ago 16h 15.2.11 9d01da634b8f c075bd047c0a osd.1 c0n01 host is offline 16h ago 16h 15.2.11 9d01da634b8f 616aeda28504 osd.2 c0n03 host is offline 16h ago 16h 15.2.11 9d01da634b8f b36453730c83 osd.3 c0n00 running (56m) 4m ago 16h 15.2.11 9d01da634b8f e043abf53206 prometheus.c0n00 c0n00 running (56m) 4m ago 16h 2.18.1 de242295e225 7cb50c04e26a After some digging into daemon logs we found Tracebacks (please see below). We also noticed that we successfully reach each host per ssh -F .... !!! We've done tcpdumps while upgrading and every SYN gets its SYNACK... ;-) Because we get no errors while deploying fresh Octopus cluster by cephadm (from https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm and cephadm prepare host is always OK) it might be a missing Python Lib or something that's not checked cephadm itself? Thank you for any hint. Christoph Ackermann Traceback: Traceback (most recent call last): File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 48, in bootstrap_exec s = io.read(1) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 402, in read raise EOFError("expected %d bytes, got %d" % (numbytes, len(buf))) EOFError: expected 1 bytes, got 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1166, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1202, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 34, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 44, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 134, in makegateway gw = gateway_bootstrap.bootstrap(io, spec) File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 102, in bootstrap bootstrap_exec(io, spec) File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 53, in bootstrap_exec raise HostNotFound(io.remoteaddress) execnet.gateway_bootstrap.HostNotFound: -F /tmp/cephadm-conf-61otabz_ -i /tmp/cephadm-identity-rt2nm0t4 root@c0n02 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 73, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 60, in create_from_spec_one replace_osd_ids=osd_id_claims.get(host, []), env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in create_single_host out, err, code = self._run_ceph_volume_command(host, cmd, env_vars=env_vars) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 295, in _run_ceph_volume_command error_ok=True) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1003, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1197, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Failed to connect to c0n02 (c0n02). Please make sure that the host is reachable and accepts connections using the cephadm SSH key To add the cephadm SSH key to the host: > ceph cephadm get-pub-key > ~/ceph.pub > ssh-copy-id -f -i ~/ceph.pub root@c0n02 To check that the host is reachable: > ceph cephadm get-ssh-config > ssh_config > ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key > chmod 0600 ~/cephadm_private_key > ssh -F ssh_config -i ~/cephadm_private_key root@c0n02

1 0

Host ceph version in dashboard incorrect after upgrade
by mabi 30 Apr '21

30 Apr '21

Hello, I upgraded my Octopus test cluster which has 5 hosts because one of the node (a mon/mgr node) was still on version 15.2.10 but all the others on 15.2.11. For the upgrade I used the following command: ceph orch upgrade start --ceph-version 15.2.11 The upgrade worked correctly and I did not see any errors in the logs but the host version in the ceph dashboard (under the navigation Cluster -> Hosts) still snows 15.2.10 for that specific node. The output of "ceph versions", shows that every component is on 15.2.11 as you can see below: { "mon": { "ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)": 3 }, "mgr": { "ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)": 2 }, "osd": { "ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)": 2 }, "mds": {}, "overall": { "ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)": 7 } } So why is it still stuck on 15.2.10 in the dashboard? Best regards, Mabi

2 4

Re: Nautilus 14.2.19 mon 100% CPU
by Robert LeBlanc 29 Apr '21

29 Apr '21

Good thought. The storage for the monitor data is a RAID-0 over three NVMe devices. Watching iostat, they are completely idle, maybe 0.8% to 1.4% for a second every minute or so. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Apr 8, 2021 at 7:48 PM Zizon Qiu <zzdtsv(a)gmail.com> wrote: > > Will it be related to some kind of disk issue of that mon located in,which may casually > slow down IO and further the rocksdb? > > > On Fri, Apr 9, 2021 at 4:29 AM Robert LeBlanc <robert(a)leblancnet.us> wrote: >> >> I found this thread that matches a lot of what I'm seeing. I see the >> ms_dispatch thread going to 100%, but I'm at a single MON, the >> recovery is done and the rocksdb MON database is ~300MB. I've tried >> all the settings mentioned in that thread with no noticeable >> improvement. I was hoping that once the recovery was done (backfills >> to reformatted OSDs) that it would clear up, but not yet. So any other >> ideas would be really helpful. Our MDS is functioning, but stalls a >> lot because the mons miss heartbeats. >> >> mon_compact_on_start = true >> rocksdb_cache_size = 1342177280 >> mon_lease = 30 >> mon_osd_cache_size = 200000 >> mon_sync_max_payload_size = 4096 >> >> ---------------- >> Robert LeBlanc >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 >> >> On Thu, Apr 8, 2021 at 1:11 PM Stefan Kooman <stefan(a)bit.nl> wrote: >> > >> > On 4/8/21 6:22 PM, Robert LeBlanc wrote: >> > > I upgraded our Luminous cluster to Nautilus a couple of weeks ago and >> > > converted the last batch of FileStore OSDs to BlueStore about 36 hours >> > > ago. Yesterday our monitor cluster went nuts and started constantly >> > > calling elections because monitor nodes were at 100% and wouldn't >> > > respond to heartbeats. I reduced the monitor cluster to one to prevent >> > > the constant elections and that let the system limp along until the >> > > backfills finished. There are large amounts of time where ceph commands >> > > hang with the CPU is at 100%, when the CPU drops I see a lot of work >> > > getting done in the monitor logs which stops as soon as the CPU is at >> > > 100% again. >> > >> > >> > Try reducing mon_sync_max_payload_size=4096. I have seen Frank Schilder >> > advise this several times because of monitor issues. Also recently for a >> > cluster that got upgraded from Luminous -> Mimic -> Nautilus. >> > >> > Worth a shot. >> > >> > Otherwise I'll try to look in depth and see if I can come up with >> > something smart (for now I need to go catch some sleep). >> > >> > Gr. Stefan

4 19

ceph pool size 1 for (temporary and expendable data) still using 2X storage?
by Joshua West 29 Apr '21

29 Apr '21

ceph pool size 1 for (temporary and expendable data) still using 2X storage? Hey Ceph Users! With all the buzz around chia coin, I want to dedicate a few TB to storage mining, really just to play with the chia CLI tool, and learn how it all works. At the whole concept is about dedicating disk space for large calculation outputs, the data is meaningless. For this reason, I am hoping to use a pool with size 1, min_size 1, and did set up the same. However, as a proxmox user, I noticed that this pool appears to still use 2X storage space, or at a minimum, the pool's maximum size is limited to 50% of total storage space (not that I plan on maxing out my storage for this.) I suspect there is a novice-user failsafe which ensures foolishly configured size=1 is automatically treated as size=2... Can anyone point me towards how best to leverage my ceph cluster to store expendable data at size=1 without wasting x2 actual disk space? My cluster is perfectly balanced, so I am reluctant to pull an osd out, generally don't have any other disks on hand, and don't plan to spend money on additional storage for this endeavour. I do want to ensure I am not wasting more space than I am expecting though. (Small hobby cluster, if it matters) Josh

1 0

Re: [ CEPH ANSIBLE FAILOVER TESTING ] Ceph Native Driver issue
by Lokendra Rathour 29 Apr '21

29 Apr '21

Hi Reed, Thankyou so much for the input and support. We have tried using the variable suggested by you, but could not see any impact on the current system. *"ceph fs set cephfs allow_standby_replay true " *it did not create any *impact in the failover time* Furthermore we have tried more scenarios that we tested using our test : *scenario 1:* [image: image.png] - In this we have tried to see the logs at the new node on which the mds will failover to, i.e in this case if we reboot cephnode2 so new active MDS will be Cephnode1. Checking logs for cephnode1 in two scenarios: - 1. *normal reboot of Cephnode2 by keeping the I/O operation in progress,* - we see that log at cephnode1 instantiates immediately and then wait for sometime (around 15 seconds for some beacon time) + some additional 6-7 seconds during which it activated MDS on cephnode1 and resumes I/O. Refer logs as : - 2021-04-29T15:49:42.480+0530 7fa747690700 1 mds.cephnode1 Updating MDS map to version 505 from mon.2 2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 handle_mds_map i am now mds.0.505 2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 handle_mds_map state change up:boot --> up:replay 2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 replay_start 2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 recovery set is 2021-04-29T15:49:42.482+0530 7fa747690700 1 mds.0.505 waiting for osdmap 486 (which blacklists prior instance) 2021-04-29T15:49:55.686+0530 7fa74568c700 1 mds.beacon.cephnode1 MDS connection to Monitors appears to be laggy; 15.9769s since last acked beacon 2021-04-29T15:49:55.686+0530 7fa74568c700 1 mds.0.505 skipping upkeep work because connection to Monitors appears laggy 2021-04-29T15:49:57.533+0530 7fa749e95700 0 mds.beacon.cephnode1 MDS is no longer laggy 2021-04-29T15:49:59.599+0530 7fa740e83700 0 mds.0.cache creating system inode with ino:0x100 2021-04-29T15:49:59.599+0530 7fa740e83700 0 mds.0.cache creating system inode with ino:0x1 2021-04-29T15:50:00.456+0530 7fa73f680700 1 mds.0.505 Finished replaying journal 2021-04-29T15:50:00.456+0530 7fa73f680700 1 mds.0.505 making mds journal writeable 2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.cephnode1 Updating MDS map to version 506 from mon.2 2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 handle_mds_map i am now mds.0.505 2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 handle_mds_map state change up:replay --> up:reconnect 2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 reconnect_start 2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.505 reopen_log 2021-04-29T15:50:00.959+0530 7fa747690700 1 mds.0.server reconnect_clients -- 2 sessions 2021-04-29T15:50:00.964+0530 7fa747690700 0 log_channel(cluster) log [DBG] : reconnect by client.6892 v1:10.0.4.96:0/1646469259 after 0.00499997 2021-04-29T15:50:00.972+0530 7fa747690700 0 log_channel(cluster) log [DBG] : reconnect by client.6990 v1:10.0.4.115:0/2776266880 after 0.0129999 2021-04-29T15:50:00.972+0530 7fa747690700 1 mds.0.505 reconnect_done 2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.cephnode1 Updating MDS map to version 507 from mon.2 2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.0.505 handle_mds_map i am now mds.0.505 2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.0.505 handle_mds_map state change up:reconnect --> up:rejoin 2021-04-29T15:50:02.005+0530 7fa747690700 1 mds.0.505 rejoin_start 2021-04-29T15:50:02.008+0530 7fa747690700 1 mds.0.505 rejoin_joint_start 2021-04-29T15:50:02.040+0530 7fa740e83700 1 mds.0.505 rejoin_done 2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.cephnode1 Updating MDS map to version 508 from mon.2 2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 handle_mds_map i am now mds.0.505 2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 handle_mds_map state change up:rejoin --> up:clientreplay 2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 recovery_done -- successful recovery! 2021-04-29T15:50:03.050+0530 7fa747690700 1 mds.0.505 clientreplay_start 2021-04-29T15:50:03.094+0530 7fa740e83700 1 mds.0.505 clientreplay_done 2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.cephnode1 Updating MDS map to version 509 from mon.2 2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.0.505 handle_mds_map i am now mds.0.505 2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.0.505 handle_mds_map state change up:clientreplay --> up:active 2021-04-29T15:50:04.081+0530 7fa747690700 1 mds.0.505 active_start 2021-04-29T15:50:04.085+0530 7fa747690700 1 mds.0.505 cluster recovered. - *hard reset/power-off of Cephnode2 by keeping the I/O operation in progress:* - In this case we see that the system logs at cephnode 1(on which new MDS will be activated) gets activated after 15+ seconds of power-off. - Time at which power-off was it : 2021-04-29-16-17-37 - *Time at which the logs started to show in cephnode 1* (refer logs) i.e log started nearly after 15 seconds of hardware reset: - 2021-04-29T16:17:51.983+0530 7f5ba3a38700 1 mds.cephnode1 Updating MDS map to version 518 from mon.0 2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map i am now mds.0.518 2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map state change up:boot --> up:replay 2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518 replay_start 2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518 recovery set is 2021-04-29T16:17:51.984+0530 7f5ba3a38700 1 mds.0.518 waiting for osdmap 504 (which blacklists prior instance) 2021-04-29T16:17:54.044+0530 7f5b9ca2a700 0 mds.0.cache creating system inode with ino:0x100 2021-04-29T16:17:54.045+0530 7f5b9ca2a700 0 mds.0.cache creating system inode with ino:0x1 2021-04-29T16:17:55.025+0530 7f5b9ba28700 1 mds.0.518 Finished replaying journal 2021-04-29T16:17:55.025+0530 7f5b9ba28700 1 mds.0.518 making mds journal writeable 2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.cephnode1 Updating MDS map to version 519 from mon.0 2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map i am now mds.0.518 2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map state change up:replay --> up:reconnect 2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518 reconnect_start 2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.518 reopen_log 2021-04-29T16:17:56.060+0530 7f5ba3a38700 1 mds.0.server reconnect_clients -- 2 sessions 2021-04-29T16:17:56.068+0530 7f5ba3a38700 0 log_channel(cluster) log [DBG] : reconnect by client.6990 v1: 10.0.4.115:0/2776266880 after 0.00799994 2021-04-29T16:17:56.069+0530 7f5ba3a38700 0 log_channel(cluster) log [DBG] : reconnect by client.6892 v1: 10.0.4.96:0/1646469259 after 0.00899994 2021-04-29T16:17:56.069+0530 7f5ba3a38700 1 mds.0.518 reconnect_done 2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.cephnode1 Updating MDS map to version 520 from mon.0 2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map i am now mds.0.518 2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map state change up:reconnect --> up:rejoin 2021-04-29T16:17:57.099+0530 7f5ba3a38700 1 mds.0.518 rejoin_start 2021-04-29T16:17:57.103+0530 7f5ba3a38700 1 mds.0.518 rejoin_joint_start 2021-04-29T16:17:57.472+0530 7f5b9d22b700 1 mds.0.518 rejoin_done 2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.cephnode1 Updating MDS map to version 521 from mon.0 2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map i am now mds.0.518 2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map state change up:rejoin --> up:clientreplay 2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518 recovery_done -- successful recovery! 2021-04-29T16:17:58.138+0530 7f5ba3a38700 1 mds.0.518 clientreplay_start 2021-04-29T16:17:58.157+0530 7f5b9d22b700 1 mds.0.518 clientreplay_done 2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.cephnode1 Updating MDS map to version 522 from mon.0 2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map i am now mds.0.518 2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.0.518 handle_mds_map state change up:clientreplay --> up:active 2021-04-29T16:17:59.178+0530 7f5ba3a38700 1 mds.0.518 active_start 2021-04-29T16:17:59.181+0530 7f5ba3a38700 1 mds.0.518 cluster recovered. *In both the test cases above* we saw some extra delay of* around 15 seconds* + 8-10 seconds. (total 21-25 seconds for failover in case of power-off/reboot), *Query:* Any specific config that may need to be tweaked/tried to reduce this time for MDS to know that it has to activate and start the standby MDS Node?) *Scenario 2:* - *Only stop MDS Daemon Service on Active Node* - In this scenario when we only tried stopping systemctl service for the MDS Node on Active Node, we have very good *reading of around 5-7 Seconds* for failover. - Deployment Mode CEPH MDS Setup Test Case I/O Resume Duration (Seconds) Node affected - 2 Node MDS Setupwith max_mds=1 Active-Standby MDS with Active Node MDS Demon stop 5-7 cephnode 1 Please suggest/advice if we can try to configure to achieve minimal failover duration in the first two scenarios. Best Regards, Lokendra On Thu, Apr 29, 2021 at 1:47 AM Reed Dier <reed.dier(a)focusvq.com> wrote: > I don't have anything of merit to add to this, but it would be an > interesting addition to your testing to see if active+standby-replay makes > any difference with test-case1. > > I don't think it would be applicable to any of the other use-cases, as a > standby-replay MDS is bound to a single rank, meaning its bound to a single > active MDS, and can't function as a standby for active:active. > > https://docs.ceph.com/en/latest/cephfs/standby/#configuring-standby-replay > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/c… > > Good luck and look forward to hearing feedback/more results. > > Reed > > On Apr 27, 2021, at 8:40 AM, Lokendra Rathour <lokendrarathour(a)gmail.com> > wrote: > > Hi Team, > We have setup two Node Ceph Cluster using *Native Cephfs Driver* with *Details > as:* > > - 3 Node / 2 Node MDS Cluster > - 3 Node Monitor Quorum > - 2 Node OSD > - 2 Nodes for Manager > > > Cephnode3 have only Mon and MDS (only for test case 4-7) rest two nodes > i.e. cephnode1 and cephnode2 have (mgr,mds,mon,rgw) > > > We have tested following failover scenarios for Native Cephfs Driver by > mounting for any one sub-volume on a VM or client with continuous I/O > operations(Directory creation after every 1 Second)*:* > > <image.png> > > > In the table above we have few queries as: > > - Refer test case 2 and test case 7, both are similar test case with > only difference in number of Ceph MDS with time for both the test cases is > different. It should be zero. But time is coming as 17 seconds for testcase > 7. > - Is there any configurable parameter/any configuration which we need > to make in the Ceph cluster to get the failover time reduced to few > seconds? > > In current default deployment we are getting something around 35-40 > seconds. > > > > > > > Best Regards, > > -- > ~ Lokendra > www.inertiaspeaks.com > www.inertiagroups.com > skype: lokendrarathour > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > -- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

1 0

ceph export not producing file?
by Piotr Baranowski 29 Apr '21

29 Apr '21

Hey all! rbd export images/ec8a7ff8-6609-4b7d-8bdd-fadcf3b7973e /root/foo.img DOES NOT produce the target file no matter if I use --pool --image format or the one above the target file is not there. Progress bar shows up and prints percentage. It ends up with exit 0 [root@controller-0 mnt]# rbd export --pool=images db8290c3-93fd-4a4e- ad71-7c131070ad6f /mnt/cirros2.img Exporting image: 100% complete...done. [root@controller-0 mnt]# echo $? 0 Any idea what's going on here? It's ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) luminous (stable) Any hints will be much appreciated best regards Piotr

2 2

one of 3 monitors keeps going down
by Robert W. Eckert 29 Apr '21

29 Apr '21

Hi, On a daily basis, one of my monitors goes down [root@cube ~]# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum rhel1.robeckert.us,story [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon mon.cube on cube.robeckert.us is in error state [WRN] MON_DOWN: 1/3 mons down, quorum rhel1.robeckert.us,story mon.cube (rank 2) addr [v2:192.168.2.142:3300/0,v1:192.168.2.142:6789/0] is down (out of quorum) [root@cube ~]# ceph --version ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) I have a script that will copy the mon data from another server and it restarts and runs well for a while. It is always the same monitor, and when I look at the logs the only thing I really see is the cephadm log showing it down 2021-04-28 10:07:26,173 DEBUG Running command: /usr/bin/podman --version 2021-04-28 10:07:26,217 DEBUG /usr/bin/podman: stdout podman version 2.2.1 2021-04-28 10:07:26,222 DEBUG Running command: /usr/bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867-osd.2 2021-04-28 10:07:26,326 DEBUG /usr/bin/podman: stdout fab17e5242eb4875e266df19ca89b596a2f2b1d470273a99ff71da2ae81eeb3c,docker.io/ceph/ceph:v15,5b724076c58f97872fc2f7701e8405ec809047d71528f79da452188daf2af72e,2021-04-26 17:13:15.54183375 -0400 EDT, 2021-04-28 10:07:26,328 DEBUG Running command: systemctl is-enabled ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube<mailto:ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867@mon.cube> 2021-04-28 10:07:26,334 DEBUG systemctl: stdout enabled 2021-04-28 10:07:26,335 DEBUG Running command: systemctl is-active ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867(a)mon.cube<mailto:ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867@mon.cube> 2021-04-28 10:07:26,340 DEBUG systemctl: stdout failed 2021-04-28 10:07:26,340 DEBUG Running command: /usr/bin/podman --version 2021-04-28 10:07:26,395 DEBUG /usr/bin/podman: stdout podman version 2.2.1 2021-04-28 10:07:26,402 DEBUG Running command: /usr/bin/podman inspect --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index .Config.Labels "io.ceph.version"}} ceph-fe3a7cb0-69ca-11eb-8d45-c86000d08867-mon.cube 2021-04-28 10:07:26,526 DEBUG /usr/bin/podman: stdout 04e7c673cbacf5160427b0c3eb2f0948b2f15d02c58bd1d9dd14f975a84cfc6f,docker.io/ceph/ceph:v15,5b724076c58f97872fc2f7701e8405ec809047d71528f79da452188daf2af72e,2021-04-28 08:54:57.614847512 -0400 EDT, I don't know if it matters, but this server is an AMD 3600XT while my other two servers which have had no issues are intel based. The root file system was originally on a SSD, and I switched to NVME, so I eliminated controller or drive issues. (I didn't see anything in dmesg anyway) If someone could point me in the right direction on where to troubleshoot next, I would appreciate it. Thanks, Rob Eckert

3 2

librbd::operation::FlattenRequest
by Lázár Imre 29 Apr '21

29 Apr '21

Hello, I cannot flatten an image, it always restarts with: root@sm-node1.in.illusion.hu:~# rbd flatten vm-hdd/vm-104-disk-1 Image flatten: 28% complete...2021-04-29 10:50:27.373 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c4009db0 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 26% complete...2021-04-29 10:50:33.053 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c4008fc0 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 0% complete...2021-04-29 10:50:34.829 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c445b470 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 39% complete...2021-04-29 10:50:42.081 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c40324e0 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 0% complete...2021-04-29 10:50:43.897 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c4018890 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 42% complete...2021-04-29 10:51:07.813 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c402fe80 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 42% complete...2021-04-29 10:51:29.372 7ff7caffd700 -1 librbd::operation::FlattenRequest: 0x7ff7c40017c0 should_complete: encountered error: (85) Interrupted system call should be restarted root@sm-node1.in.illusion.hu:~# uname -a Linux sm-node1.in.illusion.hu 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100) x86_64 GNU/Linux root@sm-node1.in.illusion.hu:~# dpkg -l |grep ceph ii ceph 14.2.20-pve1 amd64 distributed storage and file system ii ceph-base 14.2.20-pve1 amd64 common ceph daemon libraries and management tools ii ceph-common 14.2.20-pve1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-fuse 14.2.20-pve1 amd64 FUSE-based client for the Ceph distributed file system ii ceph-mds 14.2.20-pve1 amd64 metadata server for the ceph distributed file system ii ceph-mgr 14.2.20-pve1 amd64 manager for the ceph distributed storage system ii ceph-mgr-dashboard 14.2.20-pve1 all dashboard plugin for ceph-mgr ii ceph-mon 14.2.20-pve1 amd64 monitor server for the ceph storage system ii ceph-osd 14.2.20-pve1 amd64 OSD server for the ceph storage system ii libcephfs2 14.2.20-pve1 amd64 Ceph distributed file system client library ii python-ceph-argparse 14.2.20-pve1 all Python 2 utility libraries for Ceph CLI ii python-cephfs 14.2.20-pve1 amd64 Python 2 libraries for the Ceph libcephfs library root@sm-node1.in.illusion.hu:~# dpkg -l |grep rbd ii librbd1 14.2.20-pve1 amd64 RADOS block device client library ii python-rbd 14.2.20-pve1 amd64 Python 2 libraries for the Ceph librbd library root@sm-node1.in.illusion.hu:~# modinfo rbd filename: /lib/modules/5.4.106-1-pve/kernel/drivers/block/rbd.ko license: GPL description: RADOS Block Device (RBD) driver author: Jeff Garzik <jeff(a)garzik.org> author: Yehuda Sadeh <yehuda(a)hq.newdream.net> author: Sage Weil <sage(a)newdream.net> author: Alex Elder <elder(a)inktank.com> srcversion: 7BA6FEE20249E416B2D09AB depends: libceph retpoline: Y intree: Y name: rbd vermagic: 5.4.106-1-pve SMP mod_unload modversions parm: single_major:Use a single major number for all rbd devices (default: true) (bool) Please give me advice, how can I produce more information to catch this. Thank you, i.

1 0

Unable to add osds with ceph-volume
by Andrei Mikhailovsky 29 Apr '21

29 Apr '21

Hello everyone, I am running ceph version 15.2.8 on Ubuntu servers. I am using bluestore osds with data on hdd and db and wal on ssd drives. Each ssd has been partitioned such that it holds 5 dbs and 5 wals. The ssd were were prepared a while back probably when I was running ceph 13.x. I have been gradually adding new osd drives as needed. Recently, I've tried to add more osds, which have failed to my surprise. Previously I've had no issues adding the drives. However, it seems that I can no longer do that with version 15.2.x Here is what I get: root@arh-ibstorage4-ib  /home/andrei  ceph-volume lvm prepare --bluestore --data /dev/sds --block.db /dev/ssd3/db5 --block.wal /dev/ssd3/wal5 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 6aeef34b-0724-4d20-a10b-197cab23e24d Running command: /usr/sbin/vgcreate --force --yes ceph-1c7cef26-327a-4785-96b3-dcb1b97e8e2f /dev/sds stderr: WARNING: PV /dev/sdp in VG ceph-bc7587b5-0112-4097-8c9f-4442e8ea5645 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdo in VG ceph-33eda27c-53ed-493e-87a8-39e1862da809 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdn in VG ssd2 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdm in VG ssd1 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdj in VG ceph-9d8da00c-f6b9-473f-b499-fa60d74b46c5 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdi in VG ceph-1603149e-1e50-4b86-a360-1372f4243603 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdh in VG ceph-a5f4416c-8e69-4a66-a884-1d1229785acb is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sde in VG ceph-aac71121-e308-4e25-ae95-ca51bca7aaff is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdd in VG ceph-1e216580-c01b-42c5-a10f-293674a55c4c is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdc in VG ceph-630f7716-3d05-41bb-92c9-25402e9bb264 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sdb in VG ceph-a549c28d-9b06-46d5-8ba3-3bd99ff54f57 is using an old PV header, modify the VG to update. stderr: WARNING: PV /dev/sda in VG ceph-70943bd0-de71-4651-a73d-c61bc624755f is using an old PV header, modify the VG to update. stdout: Physical volume "/dev/sds" successfully created. stdout: Volume group "ceph-1c7cef26-327a-4785-96b3-dcb1b97e8e2f" successfully created Running command: /usr/sbin/lvcreate --yes -l 3814911 -n osd-block-6aeef34b-0724-4d20-a10b-197cab23e24d ceph-1c7cef26-327a-4785-96b3-dcb1b97e8e2f stdout: Logical volume "osd-block-6aeef34b-0724-4d20-a10b-197cab23e24d" created. --> blkid could not detect a PARTUUID for device: /dev/ssd3/wal5 --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.15 --yes-i-really-mean-it stderr: 2021-04-28T20:05:52.290+0100 7f76bbfa9700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc /ceph/keyring.bin,: (2) No such file or directory 2021-04-28T20:05:52.290+0100 7f76bbfa9700 -1 AuthRegistry(0x7f76b4058e60) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyrin g,/etc/ceph/keyring.bin,, disabling cephx stderr: purged osd.15 --> RuntimeError: unable to use device I have tried to find a solution, but wasn't able to resolve the problem? I am sure that I've previously added new volumes using the above command. lvdisplay shows: --- Logical volume --- LV Path /dev/ssd3/wal5 LV Name wal5 VG Name ssd3 LV UUID WPQJs9-olAj-ACbU-qnEM-6ytu-aLMv-hAABYy LV Write Access read/write LV Creation host, time arh-ibstorage4-ib, 2020-07-29 23:45:17 +0100 LV Status available # open 0 LV Size 1.00 GiB Current LE 256 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:6 --- Logical volume --- LV Path /dev/ssd3/db5 LV Name db5 VG Name ssd3 LV UUID FVT2Mm-a00P-eCoQ-FZAf-AulX-4q9r-PaDTC6 LV Write Access read/write LV Creation host, time arh-ibstorage4-ib, 2020-07-29 23:46:01 +0100 LV Status available # open 0 LV Size 177.00 GiB Current LE 45312 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:11 How do I resolve the errors and create the new osd? Cheers Andrei

3 2

active+recovery_unfound+degraded in Pacific
by Lomayani S. Laizer 29 Apr '21

29 Apr '21

Hello, Last week I upgraded my production cluster to Pacific. the cluster was healthy until a few hours ago. When scrub run 4hrs ago left the cluster in an inconsistent state. Then issued the command ceph pg repair 7.182 to try to repair the cluster but ended with active+recovery_unfound+degraded All OSDs are up and all running bluestore with replication of 3 and minimum size of 2. I have restarted all OSD but still not helping. Any recommendations on how to recover the cluster safely? I have attached result of ceph pg 7.182 query ceph health detail HEALTH_ERR 1/2459601 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 3/7045706 objects degraded (0.000%), 1 pg degraded [WRN] OBJECT_UNFOUND: 1/2459601 objects unfound (0.000%) pg 7.182 has 1 unfound objects [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound pg 7.182 is active+recovery_unfound+degraded, acting [15,1,11], 1 unfound [WRN] PG_DEGRADED: Degraded data redundancy: 3/7045706 objects degraded (0.000%), 1 pg degraded pg 7.182 is active+recovery_unfound+degraded, acting [15,1,11], 1 unfound ceph -w cluster: id: 4b9f6959-fead-4ada-ac58-de5d7b149286 health: HEALTH_ERR 1/2459586 objects unfound (0.000%) Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 3/7045661 objects degraded (0.000%), 1 pg degraded services: mon: 3 daemons, quorum mon-a,mon-b,mon-c (age 38m) mgr: mon-a(active, since 38m) osd: 46 osds: 46 up (since 25m), 46 in (since 3w) data: pools: 4 pools, 705 pgs objects: 2.46M objects, 9.1 TiB usage: 24 TiB used, 95 TiB / 119 TiB avail pgs: 3/7045661 objects degraded (0.000%) 1/2459586 objects unfound (0.000%) 701 active+clean 3 active+clean+scrubbing+deep 1 active+recovery_unfound+degraded ceph pg 7.182 list_unfound { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "rbd_data.2f18f2a67fad72.000000000002021a", "key": "", "snapid": -2, "hash": 3951004034, "max": 0, "pool": 7, "namespace": "" }, "need": "184249'118613008", "have": "0'0", "flags": "none", "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", "locations": [] } ], "state": "NotRecovering", "available_might_have_unfound": true, "might_have_unfound": [], "more": false }

3 5

Unable to delete versioned bucket
by Mark Schouten 29 Apr '21

29 Apr '21

Hi, I have a bucket that has versioning enabled and I am trying to remove it. This is not possible, because the bucket is not empty: mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api delete-bucket --bucket syslog_tuxis_net An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: Unknown It took me a while to determine that `curator` had enabled bucket versioning, but after I did I removed all versions. But, when trying to delete the bucket, it still doesn't work because there are DeleteMarkers, eg: { "Owner": { "DisplayName": "Syslog Backup", "ID": "DB0220$syslog_backup" }, "Key": "incompatible-snapshots", "VersionId": "noeJtvBpV5HINGQTJeXEq5mzlzsWneg", "IsLatest": true, "LastModified": "2021-03-22T16:35:18.298Z" }, I cannot download that object: mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api get-object --bucket syslog_tuxis_net --key incompatible-snapshots /tmp/foobar An error occurred (NoSuchKey) when calling the GetObject operation: Unknown Nor can I delete the object: mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api delete-object --bucket syslog_tuxis_net --key incompatible-snapshots So, according to https://ceph.io/planet/on-ceph-rgw-s3-object-versioning/#on-delete-marker I should be able to delete that DeleteMarker with this command: mark@tuxis:~$ aws --endpoint=https://nl.dadup.eu s3api delete-object --bucket syslog_tuxis_net --key incompatible-snapshots --version-id noeJtvBpV5HINGQTJeXEq5mzlzsWneg But that command does not give any output, and it does not delete the marker either. So I'm stuck with that bucket which I would like to remove without abusing radosgw-admin. This cluster is running 12.2.13 with civetweb rgw's behind a haproxy setup. All is working fine, except for this versioning bucket. Can anywone point me in the right direction to remove this bucket as a normal user? Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

2 3

Double slashes in s3 name
by Gavin Chen 29 Apr '21

29 Apr '21

Hello, We’ve got some issues when uploading s3 objects with a double slash // in the name, and was wondering if anyone else has observed this issue with uploading objects to the radosgw? When connecting to the cluster to upload an object with the key ‘test/my//bucket’ the request returns with a 403 (SignatureDoesNotMatch) error. Wondering if anyone else has observed this behavior and has any workarounds to work with double slashes in the object key name. - Gavin

2 1

recovering damaged rbd volume
by mike brown 28 Apr '21

28 Apr '21

hello all, I faced an incident for one of my very important rbd volumes with 5TB data, which is managed by OpenStack. I was about to increase the volume size live but I shrinked the volume unintentionally by running a wrong command of "virsh qemu-monitor-command". then I realized it and expand it again, but obviously I lost my data. can you help me or give me hints on how I can recover data of this rbd volume? unfortunately, I don't have any backup for this part of data, and it's really important (I know I made a big mistake). also, I can't stop the cluster since it's under heavy production load. It seems qemu has been using the old set of APIs that allows shrinking by default without any warnings even in the latest version. (All other standard ways of resizing volume, does not allow shrinking) * Ceph: https://sourcegraph.com/github.com/ceph/ceph@luminous/-/blob/src/librbd/lib… * Qemu: https://sourcegraph.com/github.com/qemu/qemu/-/blob/block/rbd.c#L832 * Libvirt: https://sourcegraph.com/github.com/libvirt/libvirt/-/blob/src/storage/stora…

1 0

PG repair leaving cluster unavailable
by Gesiel Galvão Bernardes 28 Apr '21

28 Apr '21

Hi, I have 3 pools, where I use it exclusively for RBD images. 2 They are mirrored and one is an erasure code. It turns out that today I received the warning that a PG was inconsistent in the pool erasure, and then I ran "ceph pg repair <pg>". It turns out that after that the entire cluster became extremely slow, to the point that no VM works. This is the output of "ceph -s": # ceph -s cluster: id: 4ea72929-6f9e-453a-8cd5-bb0712f6b874 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent, 1 pg repair services: mon: 2 daemons, cmonitor quorum, cmonitor2 mgr: cmonitor (active), standbys: cmonitor2 osd: 87 osds: 87 up, 87 in tcmu-runner: 10 active daemons date: pools: 7 pools, 3072 pgs objects: 30.00 M objects, 113 TiB usage: 304 TiB used, 218 TiB / 523 TiB avail pgs: 3063 active + clean 8 active + clean + scrubbing + deep 1 active + clean + scrubbing + deep + inconsistent + repair io: client: 24 MiB / s rd, 23 MiB / s wr, 629 op / s rd, 519 op / s wr cache: 5.9 MiB / s flush, 35 MiB / s evict, 9 op / s promote Does anyone have any idea how to make it available again? Regards, Gesiel

1 1

Re: Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
by Szabo, Istvan (Agoda) 28 Apr '21

28 Apr '21

Hello, Thank you very much to pickup the question and sorry for the late response. Yes, we are sending in cleartext also using HTTPS, but how it should be send if not like this? Also connected to this issue a bit, when we subscribe a bucket to a topic with non-ACL kafka topic, any operations (PUT or DELETE) is simply blocking and not returning. Not even any error response. $ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3 certificate.pdf s3://vig-test WARNING: certificate.pdf: Owner groupname not known. Storing GID=1354917867 instead. WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf' [1 of 1] 65536 of 91224 71% in 0s 291.17 KB/s Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Yuval Lifshitz <ylifshit(a)redhat.com> Sent: Wednesday, April 21, 2021 10:34 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint Hi Istvan, Can you please share the relevant part for the radosgw log, indicating which input was invalid? The only way I managed to reproduce that error is by sending the request to a non-HTTPS radosgw (which does not seem to be your case). In such a case it replies with "InvalidInput" because we are trying to send user/password in cleartext. I used curl, similarly to what you did against a vstart cluster based off of master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM Yuval On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: Hi Ceph Users, Here is the latest request I tried but still not working curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<password>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx000000000000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-data</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com><mailto:istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

2 3

BlueFS.cc ceph_assert(bl.length() <= runway): protection against bluefs log file growth
by Konstantin Shalygin 28 Apr '21

28 Apr '21

Hi, Recently was added [1] protection against BlueFS log growth infinite, I get assert on 14.2.19: /build/ceph-14.2.19/src/os/bluestore/BlueFS.cc: 2404: FAILED ceph_assert(bl.length() <= runway) Then OSD dead. Tracker (may be already exists?), logs is interested for this case? [1] https://github.com/ceph/ceph/pull/37948 <https://github.com/ceph/ceph/pull/37948> Thanks, k

1 0

Cephadm multiple public networks
by Stanislav Datskevych 28 Apr '21

28 Apr '21

Hi all, I've installed latest Pacific version 16.2.1 using Cephadm. I try using multiple public networks with this setting: ceph config set mon public_network "100.90.1.0/24,100.90.2.0/24" The networks seem to be successfully passed to /etc/ceph/ceph.conf on the daemons, however I constantly see the following messages in the log: 4/26/21 11:40:05 PM[INF]Filtered out host mon-1: could not verify host allowed virtual ips 4/26/21 11:40:05 PM[INF]Filtered out host mon-2: could not verify host allowed virtual ips 4/26/21 11:40:05 PM[INF]Filtered out host mon-3: could not verify host allowed virtual ips As the result, ceph orch doesn't deploy any monitor. Eventually I was able to deploy monitors by first setting public_network to 100.90.1.0/24, then deploy monitors in this subnet, then setting public_network to 100.90.2.0/24, deploy the rest of monitors. Is this some kind of bug or I miss something? Thanks for the replies in advance.

4 5

Rbd map fails occasionally with module libceph: Relocation (type 6) overflow vs section 4
by huxiaoyu＠horebdata.cn 28 Apr '21

28 Apr '21

Dear Cephers, I encountered a strange issue when using rbd map (Luminous 12.2.13 version), rbd map not always fail, but occasionally, with the following dmesg [16818.700000] module libceph: Relocation (type 6) overflow vs section 4 [16857.460000] module libceph: Relocation (type 6) overflow vs section 4 [16891.850000] module libceph: Relocation (type 6) overflow vs section 4 what could be wrong? and how to fix it? Any help or suggestions would be highly appreciated, thanks a lot in advance, samuel huxiaoyu(a)horebdata.cn

1 0

How to set bluestore_rocksdb_options_annex
by ceph＠elchaka.de 28 Apr '21

28 Apr '21

Hello, I have an octopus cluster and want to change some values - but i cannot find any documentation on how to set values(multiple) with bluestore_rocksdb_options_annex Could someone give me some examples. I would like to do this like ceph config set ... Thanks in advice Mehmet

1 0

ceph-csi on openshift
by Bosteels Nino 27 Apr '21

27 Apr '21

Hi, I’m trying to get the ceph-csi working on openshift (I followed this guide: https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/) On openshift it seems you can’t run privileged containers per default and can’t use HostPath etc. For these you need to create a security context constraint (a custom one). I’d like to enable the next person that searches for this, so I contacted red hat through our support plan, and they suggested: kind: SecurityContextConstraints apiVersion: v1 metadata: name: custom-scc allowPrivilegedContainer: true allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowedCapabilities: - KILL - NET_ADMIN - SYS_ADMIN - SYS_BOOT - SYS_TIME runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny fsGroup: type: RunAsAny supplementalGroups: type: RunAsAny users: - <your-user-for-which-the-previleges-are-required> It didn’t work to create this after going through the guide, so I’ll run through it again, but wanted to ask if anyone else has already done this ánd also if someone could add it to the ceph wiki. Kr, Nino *************************************************************** Dit e-mail bericht inclusief eventuele ingesloten bestanden kan informatie bevatten die vertrouwelijk is en/of beschermd door intellectuele eigendomsrechten. Dit bericht is uitsluitend bestemd voor de geadresseerde(n). Elk gebruik van de informatie vervat in dit bericht (waaronder de volledige of gedeeltelijke reproductie of verspreiding onder elke vorm) door andere personen dan de geadresseerde(n) is verboden. Indien u dit bericht per vergissing heeft ontvangen, gelieve de afzender hiervan te verwittigen en dit bericht te verwijderen. This e-mail and any attachment thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the addressees. Any use of the information contained herein (including but not limited to total or partial reproduction or distribution in any form) by other persons than the addressees is prohibited. If you have received this e-mail in error, please notify the sender and delete its contents. Ce courriel et les annexes eventuelles peuvent contenir des informations confidentielles et/ou protegees par des droits de propriete intellectuelle. Ce message est adresse exclusivement e son (ses) destinataire(s). Toute utilisation du contenu de ce message (y compris la reproduction ou diffusion partielle ou complete sous toute forme) par une autre personne que le(s) destinataire(s) est formellement interdite. Si vous avez recu ce message par erreur, veuillez prevenir l expediteur du message et en detruire le contenu. ***************************************************************e

2 3

Re: how to handle rgw leaked data (aka data that is not available via buckets but eats diskspace)
by Boris Behrens 27 Apr '21

27 Apr '21

Hi Anthony, yes we are using replication, the lost space is calculated before it's replicated. RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.1 PiB 191 TiB 968 TiB 968 TiB 83.55 TOTAL 1.1 PiB 191 TiB 968 TiB 968 TiB 83.55 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 0 64 0 B 0 0 B 0 13 TiB .rgw.root 1 64 99 KiB 119 99 KiB 0 13 TiB eu-central-1.rgw.control 2 64 0 B 8 0 B 0 13 TiB eu-central-1.rgw.data.root 3 64 947 KiB 2.82k 947 KiB 0 13 TiB eu-central-1.rgw.gc 4 64 101 MiB 128 101 MiB 0 13 TiB eu-central-1.rgw.log 5 64 267 MiB 500 267 MiB 0 13 TiB eu-central-1.rgw.users.uid 6 64 2.9 MiB 6.91k 2.9 MiB 0 13 TiB eu-central-1.rgw.users.keys 7 64 263 KiB 6.73k 263 KiB 0 13 TiB eu-central-1.rgw.meta 8 64 384 KiB 1k 384 KiB 0 13 TiB eu-central-1.rgw.users.email 9 64 40 B 1 40 B 0 13 TiB eu-central-1.rgw.buckets.index 10 64 10 GiB 67.28k 10 GiB 0.03 13 TiB eu-central-1.rgw.buckets.data 11 2048 313 TiB 151.71M 313 TiB 89.25 13 TiB ... EC profile is pretty standard [root@s3db16 ~]# ceph osd erasure-code-profile ls default [root@s3db16 ~]# ceph osd erasure-code-profile get default k=2 m=1 plugin=jerasure technique=reed_sol_van We use mainly ceph 14.2.18. There is an OSD host with 14.2.19 and one with 14.2.20 Object populations is mixed, but the most amount of data is in huge files. We store our platforms RBD snapshots in it. Cheers Boris Am Di., 27. Apr. 2021 um 06:49 Uhr schrieb Anthony D'Atri < anthony.datri(a)gmail.com>: > Are you using Replication? EC? How many copies / which profile? > On which Ceph release were your OSDs built? BlueStore? Filestore? > What is your RGW object population like? Lots of small objects? Mostly > large objects? Average / median object size? > > > On Apr 26, 2021, at 9:32 PM, Boris Behrens <bb(a)kervyn.de> wrote: > > > > HI, > > > > we still have the problem that our rgw eats more diskspace than it > should. > > Summing up the "size_kb_actual" of all buckets show only half of the used > > diskspace. > > > > There are 312TiB stored acording to "ceph df" but we only need around > 158TB. > > > > I've already wrote to this ML with the problem, but there were no > solutions > > that would help. > > I've doug through the ML archive and found some interesting threads > > regarding orphan objects and these kind of issues. > > > > Did someone ever solved this problem? > > Or do you just add more disk space. > > > > we tried to: > > * use the "radosgw-admin orphan find/finish" tool (didn't work) > > * manually triggering the GC (didn't work) > > > > currently running (since yesterday evening): > > * rgw-orphan-list, which procused 270GB of text output, and it's not done > > yet (I have 60GB diskspace left) > > > > -- > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > > groÃƒ¼en Saal. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

Re: RBD tuning for virtualization (all flash)
by by morphin 27 Apr '21

27 Apr '21

2x10G for cluster + Public 2x10G for Users lacp = 802.3ad Smart Weblications GmbH <info(a)smart-weblications.de>, 26 Nis 2021 Pzt, 17:25 tarihinde şunu yazdı: > > Hi, > > > Am 25.04.2021 um 03:58 schrieb by morphin: > > Hello. > > > > We're running 1000vm on 28 node with 6 ssd (no seperate db device) and > > these vms are Mostly win10. > > > > 2 lvm osd Per 4tb device total 288osd and one RBD pool with 8192 PG. > > Replication 3. > > > > Ceph version : nautilus 14.2.16 > > > > I'm looking for all flash RBD tuning. > > This is good test env and tomorrow gonna be prod. In vms I see 1800mb/s > > read and 900mb/s write (qemu = write back) > > With good tuning I believe this cluster will go further. > > Do you have any special suggestion? > > > > What is the network bandwidth? > > > -- > > Mit freundlichen Grüßen, > > > Smart Weblications GmbH > Martinsberger Str. 1 > D-95119 Naila > > fon.: +49 9282 9638 200 > fax.: +49 9282 9638 205 > 24/7: +49 900 144 000 00 - 0,99 EUR/Min* > http://www.smart-weblications.de > > -- > Sitz der Gesellschaft: Naila > Geschäftsführer: Florian Wiessner > HRB-Nr.: HRB 3840 Amtsgericht Hof > *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz

1 0

[ CEPH ANSIBLE FAILOVER TESTING ] Ceph Native Driver issue
by Lokendra Rathour 27 Apr '21

27 Apr '21

Hi Team, We have setup two Node Ceph Cluster using *Native Cephfs Driver* with *Details as:* - 3 Node / 2 Node MDS Cluster - 3 Node Monitor Quorum - 2 Node OSD - 2 Nodes for Manager Cephnode3 have only Mon and MDS (only for test case 4-7) rest two nodes i.e. cephnode1 and cephnode2 have (mgr,mds,mon,rgw) We have tested following failover scenarios for Native Cephfs Driver by mounting for any one sub-volume on a VM or client with continuous I/O operations(Directory creation after every 1 Second)*:* [image: image.png] In the table above we have few queries as: - Refer test case 2 and test case 7, both are similar test case with only difference in number of Ceph MDS with time for both the test cases is different. It should be zero. But time is coming as 17 seconds for testcase 7. - Is there any configurable parameter/any configuration which we need to make in the Ceph cluster to get the failover time reduced to few seconds? In current default deployment we are getting something around 35-40 seconds. Best Regards, -- ~ Lokendra www.inertiaspeaks.com www.inertiagroups.com skype: lokendrarathour

1 0

Re: ceph-volume batch does not find available block_db
by Dan van der Ster 27 Apr '21

27 Apr '21

Hi Sebastian!! (solution below) This is weird, because we had previously tested the ceph-volume refactor and it looked ok. Anyway, here is the inventory output: https://pastebin.com/ADFeuNZi And the ceph-volume log is here: https://termbin.com/i8mk I couldn't digest why it was rejected. I believe I'm using ceph-volume as intended... https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#idempotency-and-disk… Wait -- I solved my problem. The OSDs were originally created like this: ceph-volume lvm batch /dev/sd[g-z] /dev/sda[a-d] --db-devices /dev/sd[c-f] Now in order to recreate the osd on sdg, I had tried: ceph-volume lvm batch /dev/sdg --db-devices /dev/sdf --osd-id 1 That doesn't work. But, if I use all devices again, it works! # ceph-volume lvm batch /dev/sd[g-z] /dev/sda[a-d] --db-devices /dev/sd[c-f] --osd-ids 1 --> passed data devices: 24 physical, 0 LVM --> relative data size: 1.0 --> passed block_db devices: 4 physical, 0 LVM Total OSDs: 1 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- OSD id 1 data /dev/sdg 5.46 TB 100.00% block_db /dev/sdf 37.26 GB 16.67% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) There are several interesting behaviours -- if I pass fewer db-devices or HDDs it doesn't work as expected. So lesson learned, in recent ceph releases one must use ceph-volume batch exactly as it was used originally. Best Regards, Dan On Tue, Apr 27, 2021 at 2:16 PM Sebastien Han <shan(a)redhat.com> wrote: > > Hi Dan, > > I believe either the ceph-volume logs or the "ceph-volume inventory > /dev/sdf" command should give you the reason why the device was > rejected. > If not legit that's probably a bug... > > Thanks! > ––––––––– > Sébastien Han > Senior Principal Software Engineer, Storage Architect > > "Always give 100%. Unless you're giving blood." > > On Tue, Apr 27, 2021 at 2:02 PM Dan van der Ster <dan(a)vanderster.com> wrote: > > > > Hi all, > > > > In 14.2.20, when re-creating a mixed OSD after device replacement, > > ceph-volume batch is no longer able to find any available space for a > > block_db. > > > > Below I have shown a zap [1] which frees up the HDD and one LV on the > > block-dbs VG. > > But then we try to recreate, and none of the block-dbs are available > > [2], even though there is free space on the VG: > > > > VG #PV #LV #SN Attr VSize VFree > > ceph-8dfd7f83-b60c-485b-9517-12203301a914 1 5 0 wz--n- <223.57g 37.26g > > > > This bug looks similar: https://tracker.ceph.com/issues/49096 > > > > Is there something wrong with my procedure? Or does someone have an > > idea how to make this work again? > > > > Best Regards, > > > > Dan > > > > > > [1] > > > > # systemctl stop ceph-osd@1 > > # ceph osd out 1 > > marked out osd.1. > > # ceph-volume lvm zap --osd-id=1 --destroy > > --> Zapping: /dev/ceph-15daeeaa-b6d9-46a6-b955-fd7197341334/osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61 > > --> Unmounting /var/lib/ceph/osd/ceph-1 > > Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1 > > stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted > > Running command: /usr/bin/dd if=/dev/zero > > of=/dev/ceph-15daeeaa-b6d9-46a6-b955-fd7197341334/osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61 > > bs=1M count=10 conv=fsync > > stderr: 10+0 records in > > 10+0 records out > > stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0849557 s, 123 MB/s > > --> Only 1 LV left in VG, will proceed to destroy volume group > > ceph-15daeeaa-b6d9-46a6-b955-fd7197341334 > > Running command: /usr/sbin/vgremove -v -f > > ceph-15daeeaa-b6d9-46a6-b955-fd7197341334 > > stderr: Removing > > ceph--15daeeaa--b6d9--46a6--b955--fd7197341334-osd--block--ef46b9bf--c85f--49d8--9db3--9ed164f5cc61 > > (253:0) > > stderr: Archiving volume group > > "ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" metadata (seqno 5). > > stderr: Releasing logical volume > > "osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61" > > stderr: Creating volume group backup > > "/etc/lvm/backup/ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" (seqno 6). > > stdout: Logical volume > > "osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61" successfully removed > > stderr: Removing physical volume "/dev/sdg" from volume group > > "ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" > > stdout: Volume group "ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" > > successfully removed > > --> Zapping: /dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 > > Running command: /usr/bin/dd if=/dev/zero > > of=/dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 > > bs=1M count=10 conv=fsync > > stderr: 10+0 records in > > 10+0 records out > > stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0310497 s, 338 MB/s > > --> More than 1 LV left in VG, will proceed to destroy LV only > > --> Removing LV because --destroy was given: > > /dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 > > Running command: /usr/sbin/lvremove -v -f > > /dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 > > stdout: Logical volume "osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0" > > successfully removed > > stderr: Removing > > ceph--8dfd7f83--b60c--485b--9517--12203301a914-osd--db--d84267a8--057a--4b54--b1d4--7894e3eabec0 > > (253:1) > > stderr: Archiving volume group > > "ceph-8dfd7f83-b60c-485b-9517-12203301a914" metadata (seqno 25). > > stderr: Releasing logical volume "osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0" > > stderr: Creating volume group backup > > "/etc/lvm/backup/ceph-8dfd7f83-b60c-485b-9517-12203301a914" (seqno > > 26). > > --> Zapping successful for OSD: 1 > > # > > > > [2] > > > > # ceph-volume lvm batch /dev/sdg --db-devices /dev/sdf --osd-ids 1 > > --> passed data devices: 1 physical, 0 LVM > > --> relative data size: 1.0 > > --> passed block_db devices: 1 physical, 0 LVM > > --> 1 fast devices were passed, but none are available > > > > Total OSDs: 0 > > > > Type Path > > LV Size % of device > > --> The above OSDs would be created if the operation continues > > --> do you want to proceed? (yes/no) no > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >

1 0

ceph-volume batch does not find available block_db
by Dan van der Ster 27 Apr '21

27 Apr '21

Hi all, In 14.2.20, when re-creating a mixed OSD after device replacement, ceph-volume batch is no longer able to find any available space for a block_db. Below I have shown a zap [1] which frees up the HDD and one LV on the block-dbs VG. But then we try to recreate, and none of the block-dbs are available [2], even though there is free space on the VG: VG #PV #LV #SN Attr VSize VFree ceph-8dfd7f83-b60c-485b-9517-12203301a914 1 5 0 wz--n- <223.57g 37.26g This bug looks similar: https://tracker.ceph.com/issues/49096 Is there something wrong with my procedure? Or does someone have an idea how to make this work again? Best Regards, Dan [1] # systemctl stop ceph-osd@1 # ceph osd out 1 marked out osd.1. # ceph-volume lvm zap --osd-id=1 --destroy --> Zapping: /dev/ceph-15daeeaa-b6d9-46a6-b955-fd7197341334/osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61 --> Unmounting /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1 stderr: umount: /var/lib/ceph/osd/ceph-1 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-15daeeaa-b6d9-46a6-b955-fd7197341334/osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0849557 s, 123 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-15daeeaa-b6d9-46a6-b955-fd7197341334 Running command: /usr/sbin/vgremove -v -f ceph-15daeeaa-b6d9-46a6-b955-fd7197341334 stderr: Removing ceph--15daeeaa--b6d9--46a6--b955--fd7197341334-osd--block--ef46b9bf--c85f--49d8--9db3--9ed164f5cc61 (253:0) stderr: Archiving volume group "ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" metadata (seqno 5). stderr: Releasing logical volume "osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61" stderr: Creating volume group backup "/etc/lvm/backup/ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" (seqno 6). stdout: Logical volume "osd-block-ef46b9bf-c85f-49d8-9db3-9ed164f5cc61" successfully removed stderr: Removing physical volume "/dev/sdg" from volume group "ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" stdout: Volume group "ceph-15daeeaa-b6d9-46a6-b955-fd7197341334" successfully removed --> Zapping: /dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0310497 s, 338 MB/s --> More than 1 LV left in VG, will proceed to destroy LV only --> Removing LV because --destroy was given: /dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 Running command: /usr/sbin/lvremove -v -f /dev/ceph-8dfd7f83-b60c-485b-9517-12203301a914/osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0 stdout: Logical volume "osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0" successfully removed stderr: Removing ceph--8dfd7f83--b60c--485b--9517--12203301a914-osd--db--d84267a8--057a--4b54--b1d4--7894e3eabec0 (253:1) stderr: Archiving volume group "ceph-8dfd7f83-b60c-485b-9517-12203301a914" metadata (seqno 25). stderr: Releasing logical volume "osd-db-d84267a8-057a-4b54-b1d4-7894e3eabec0" stderr: Creating volume group backup "/etc/lvm/backup/ceph-8dfd7f83-b60c-485b-9517-12203301a914" (seqno 26). --> Zapping successful for OSD: 1 # [2] # ceph-volume lvm batch /dev/sdg --db-devices /dev/sdf --osd-ids 1 --> passed data devices: 1 physical, 0 LVM --> relative data size: 1.0 --> passed block_db devices: 1 physical, 0 LVM --> 1 fast devices were passed, but none are available Total OSDs: 0 Type Path LV Size % of device --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) no

1 0

Profiling/expectations of ceph reads for single-host bandwidth on fast networks?
by Erik Lindahl 27 Apr '21

27 Apr '21

Hi friends, We've recently deployed a few all-flash OSD nodes to improve both bandwidth and IOPS for active data processing in CephFS, but before taking it into active production we've been tuning it to see how far we can get the performance in practice - it would be interesting to hear your experience both about the bandwidth that's realistic to expect, and any hints on remaining profiling we could do to identify the bottlenecks. Is it possible to have rados (or even cephfs) on a single host reach anywhere close to line rate for 50Gb networking? Our setup: * We use four dedicated SSD-only OSD nodes (Dell R7515, EPYC 7302) each with 16x Samsung PM883 7.68TB enterprise SSDs connected to a H740P raid controller where each disk is configured as a RAID0 drive (we have tested HBA mode too, and as expected the battery-backed write-back caching significantly improves latency for small writes). * The client node is a slightly older supermicro dual Xeon E5-2620v4. Both the OSD nodes and clients have 128GB RAM, and CPU throttling has been disabled. * We use Mellanox 50Gb network cards, and when using iperf2 we get very close to line speed throughput between all servers after doing the usual sysconf settings to increase network buffers and increasing the card ring buggers to at least 4096. (Say ~46Gb). * All nodes have ceph pacific (16.2.0) installed through cephadm, and Linux kernel 5.8.0 as part of Ubuntu 20.04.2. All storage is bluestore. To start with plain Rados benchmarking (rados bench), the write performance for 4M blocks is quite decent with a 3-fold-replicated pool. At 16 threads we get 2.3GB/s, when bumping it to 32 threads it increases to roughly 2.8GB/s. The client load remains low during writing, and if we reduce the replicated pool size to 2 instead of 3, these numbers improve to ~3.5GB/s and ~4.2GB/s, so I assume the remaining overhead is due to latencies with the extra copies. However, those numbers are good enough that we don't really worry about it :-) However.... when it comes to reading, we seem to be stuck at around 2GB/s no matter what we try. The load on the client is also quite high, with the "rados bench" process using ~300% CPU. To test things, we decided to shut down one of the four OSD servers - which hardly has any effect on writing throughput, and none whatsoever on the read throughput. In other words, it seems the bottleneck is somewhere on the client side? Second, when we add CephFS, we lose quite another bit of performance. If we copy a single large (5GB) file between cephfs and /dev/shm (dropping page caches between trials), we see write performance of roughly 1.8GB/s, while the read performance is just 1GB/s. (For CephFS clients, we use the kernel client in Linux-5.8 with mount options noatime,nowsync,rsize=67108864,wsize=67108864,readdir_max_entries=8192,readdir_max_bytes=4194304,rasize=1073741824). While the absolute performance is quite OK, it seems a bit sad to only achieve ~35% of line rate for writes and as little as 10% for reads, so we want to make sure we're not leaving anything on the table here. Any suggestions what we could do to identify the bottlenecks would be welcome; we'd be quite happy to invest in additional hardware if necessary, but right now we're not quite sure what could be done to improve things :-) All the best, Erik -- Erik Lindahl <erik.lindahl(a)gmail.com> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden

1 0

OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
by Jonas Jelten 27 Apr '21

27 Apr '21

Hi! After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag". The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs". Tracker issue with logs: https://tracker.ceph.com/issues/50017 Anyone else encountered this error? I've "suspended" the upgrade for now :) -- Jonas

5 12

RGW bilog autotrim not working / large OMAP
by Björn Dolkemeier 27 Apr '21

27 Apr '21

Hi, we have a problem with rgw bilogs not being trimmed. This is a multisite with 2 sites on Nautilius 14.2.18, sync status is fine. radosgw-admin bilog list grows especially for buckets with lots of deletes. So from time to time we get this "large OMAP" warnings, which we easily fix with bilog trim on the affected buckets and deep-scrubbing the PG. As this is annoying I digged deeper and found these errors occurring 16 times in a batch for random buckets. It seems to affect every bucket. I think this is related to autotrimming which is not working in our case. The command "radosgw-admin bilog autotrim" shows the same messages (and does no trimming): ERROR: failed to get bucket instance info for .bucket.meta.57d082821ddc45a097b2e3401388e93d/mybucket:feec71ec-408e-4a2b-b3bd-4d214e7c8685.7724460.7 But these are buckets/instances that exists (no shards btw): radosgw-admin metadata get bucket.instance:57d082821ddc45a097b2e3401388e93d/mybucket:feec71ec-408e-4a2b-b3bd-4d214e7c8685.7724460.7 { "key": "bucket.instance:57d082821ddc45a097b2e3401388e93d/mybucket:feec71ec-408e-4a2b-b3bd-4d214e7c8685.7724460.7", "ver": { "tag": "_2pmP8x8r0exeEyQ7pn2r0nG", "ver": 1 }, "mtime": "2020-01-20 08:52:34.220312Z", "data": { "bucket_info": { "bucket": { "name": "mybucket", "marker": "feec71ec-408e-4a2b-b3bd-4d214e7c8685.7724460.7", "bucket_id": "feec71ec-408e-4a2b-b3bd-4d214e7c8685.7724460.7", "tenant": "57d082821ddc45a097b2e3401388e93d", Does that look familiar to anyone? Any ideas where to dig further? Best regards, Björn

1 0

how to handle rgw leaked data (aka data that is not available via buckets but eats diskspace)
by Boris Behrens 27 Apr '21

27 Apr '21

HI, we still have the problem that our rgw eats more diskspace than it should. Summing up the "size_kb_actual" of all buckets show only half of the used diskspace. There are 312TiB stored acording to "ceph df" but we only need around 158TB. I've already wrote to this ML with the problem, but there were no solutions that would help. I've doug through the ML archive and found some interesting threads regarding orphan objects and these kind of issues. Did someone ever solved this problem? Or do you just add more disk space. we tried to: * use the "radosgw-admin orphan find/finish" tool (didn't work) * manually triggering the GC (didn't work) currently running (since yesterday evening): * rgw-orphan-list, which procused 270GB of text output, and it's not done yet (I have 60GB diskspace left) -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

DocuBetter Meeting 1AM UTC Thursday, April 29 2021
by John Zachary Dover 26 Apr '21

26 Apr '21

A DocuBetter meeting is scheduled for later this week at 11AM AEST Thursday, which is 6PM PDT Wednesday. This meeting is not much attended, though, so unless I get responses to this email thread, I'm not going to hold it. This email is a sincere request for documentation complaints. If anything about the documentation irritates you, now's the time to tell me. If anything about the documentation is incomplete or incorrect, I'm the guy you should tell. You don't have to attend the DocuBetter Meetings to get changes into the documentation, you just have to ask me and explain clearly what needs to be changed. A couple of documentation initiatives are underway right now: I am cleaning the syntax of the whole documentation suite (a multi-month project that is as tedious as it sounds... I am currently in the middle of the cephadm documentation), and the consolidation of the Intro Guide and part of the Developer Guide as well as the beefing up of that material when it is consolidated. So: If someone responds to this email and says that they will be at the DocuBetter meeting, then I will hold the meeting. However, if by twelve hours before the time of the meeting no one has responded, there will be no DocuBetter meeting. Remember: even if no one wants to have the meeting this week, this does not mean that you can't get changes into the docs. Write to me anytime with your complaints. No complaint is too crude, no irritation will be summarily dismissed. That's it. Here are the links relevant to DocuBetter Meetings: Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 26 Apr '21

26 Apr '21

Hi Amit: Both clusters have a lot of recovering shards. Actually I do not know if it’s normal or not. 🙁 The rgw_rados_hander is the default value, I have not touched this parameter. Do I need to increase this value? Thanks Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月26日周一下午10:42写道： > Both clusters show sync status are up to date or is there any behind > shards, how many gw endpoints you configure and what rgw_rados_hander > parameters are set? > > On Mon, Apr 26, 2021 at 8:59 AM 特木勒 <twl007(a)gmail.com> wrote: > >> Another problem I notice for a new bucket, the first object in the bucket >> will not be sync. the sync will start with the second object. I tried to >> fix the index on the bucket and manually rerun bucket sync, but the first >> object still does not sync with secondary cluster. >> >> Do you have any ideas for this issue? >> >> Thanks >> >> 特木勒 <twl007(a)gmail.com> 于2021年4月26日周一上午11:16写道： >> >>> Hi Istvan: >>> >>> Thanks Amit's suggestion. >>> >>> I followed his suggestion to fix bucket index and re-do sync on buckets, >>> but it still did not work for me. >>> >>> Then I tried to use bucket rewrite command to rewrite all the objects in >>> buckets and it works for me. I think the reason is there's something wrong >>> with bucket index and rewrite has rebuilt the index. >>> >>> Here's the command I use: >>> `sudo radosgw-admin bucket rewrite -b BUCKET-NAME --min-rewrite-size 0` >>> >>> Maybe you can try this to fix the sync issues. >>> >>> @Amit Ghadge <amitg.b14(a)gmail.com> Thanks for your suggestions. Without >>> your suggestions, I will not notice something wrong with index part. >>> >>> Thanks :) >>> >>> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年4月26日周一上午9:57写道： >>> >>>> Hi, >>>> >>>> >>>> >>>> No, doesn’t work, now we will write our own sync app for ceph, I gave >>>> up. >>>> >>>> >>>> >>>> Istvan Szabo >>>> Senior Infrastructure Engineer >>>> --------------------------------------------------- >>>> Agoda Services Co., Ltd. >>>> e: istvan.szabo(a)agoda.com >>>> --------------------------------------------------- >>>> >>>> >>>> >>>> *From:* 特木勒 <twl007(a)gmail.com> >>>> *Sent:* Friday, April 23, 2021 7:50 PM >>>> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> *Cc:* ceph-users(a)ceph.io >>>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >>>> does not sync olds data >>>> >>>> >>>> >>>> Hi Istvan: >>>> >>>> >>>> >>>> We just upgraded whole cluster to 15.2.10 and the multiple site still >>>> cannot sync whole objects to secondary cluster. 🙁 >>>> >>>> >>>> >>>> Do you have any suggestions on this? And I open another issues in ceph >>>> tracker site: >>>> >>>> https://tracker.ceph.com/issues/50474 >>>> >>>> >>>> >>>> Hope someone could go to check this issue. >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> 特木勒 <twl007(a)gmail.com>于2021年3月22日周一下午9:08写道： >>>> >>>> Thank you~ >>>> >>>> >>>> >>>> I will try to upgrade cluster too. Seem like this is the only way for >>>> now. 😭 >>>> >>>> >>>> >>>> I will let you know once I complete testing. :) >>>> >>>> >>>> >>>> Have a good day >>>> >>>> >>>> >>>> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>于2021年3月22日周一下午3:38写道： >>>> >>>> Yeah, doesn't work. Last week they fixed my problem ticket which caused >>>> the crashes, and due to the crashes stopped the replication I'll give a try >>>> this week again after the update if the daemon doesn't crash, maybe it will >>>> work, because if crash hasn't happened, the data was synced. Fingers >>>> crossed ;) Don't give up 😄 >>>> ------------------------------ >>>> >>>> *From:* 特木勒 <twl007(a)gmail.com> >>>> *Sent:* Monday, March 22, 2021 1:38 PM >>>> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> *Cc:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> >>>> >>>> >>>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >>>> does not sync olds data >>>> >>>> >>>> >>>> Hi Istvan: >>>> >>>> >>>> >>>> Do you have any update on directional sync? >>>> >>>> >>>> >>>> I am trying to upgrade cluster to 15.2.10 to see if the problem is >>>> solved. :( >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年3月1日周一上午10:01写道： >>>> >>>> So-so. I had some interruption so it failed on one site, but the other >>>> is kind of working. This is the first time when I saw data caught up in the >>>> radosgw-admin data sync status on 1 side. >>>> >>>> Today will finish the other problematic site, I’ll let you know the >>>> result is it working or not. >>>> >>>> >>>> >>>> Istvan Szabo >>>> Senior Infrastructure Engineer >>>> --------------------------------------------------- >>>> Agoda Services Co., Ltd. >>>> e: istvan.szabo(a)agoda.com >>>> --------------------------------------------------- >>>> >>>> >>>> >>>> *From:* 特木勒 <twl007(a)gmail.com> >>>> *Sent:* Sunday, February 28, 2021 1:34 PM >>>> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> *Cc:* ceph-users(a)ceph.io >>>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >>>> does not sync olds data >>>> >>>> >>>> >>>> Email received from outside the company. If in doubt don't click links >>>> nor open attachments! >>>> ------------------------------ >>>> >>>> Hi Istvan: >>>> >>>> >>>> >>>> Thanks for your reply. >>>> >>>> >>>> >>>> Does directional sync solve the problem? I tried to run `radosgw-admin >>>> sync init`, bit it still did not work. :( >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年2月26日周五上午7:47写道： >>>> >>>> Same for me, 15.2.8 also. >>>> I’m trying directional sync now, looks like symmetrical has issue. >>>> >>>> Istvan Szabo >>>> Senior Infrastructure Engineer >>>> --------------------------------------------------- >>>> Agoda Services Co., Ltd. >>>> e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> >>>> --------------------------------------------------- >>>> >>>> On 2021. Feb 26., at 1:03, 特木勒 <twl007(a)gmail.com> wrote: >>>> >>>> Email received from outside the company. If in doubt don't click links >>>> nor open attachments! >>>> ________________________________ >>>> >>>> Hi all: >>>> >>>> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) >>>> >>>> I have a strange question, I just create a multiple site for Ceph >>>> cluster. >>>> But I notice the old data of source cluster is not synced. Only new data >>>> will be synced into second zone cluster. >>>> >>>> Is there anything I need to do to enable full sync for bucket or this >>>> is a >>>> bug? >>>> >>>> Thanks >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> ________________________________ >>>> This message is confidential and is for the sole use of the intended >>>> recipient(s). It may also be privileged or otherwise protected by copyright >>>> or other legal rules. If you have received it by mistake please let us know >>>> by reply email and delete it from your system. It is prohibited to copy >>>> this message or disclose its content to anyone. Any confidentiality or >>>> privilege is not waived or lost by any mistaken delivery or unauthorized >>>> disclosure of the message. All messages sent to and from Agoda may be >>>> monitored to ensure compliance with company policies, to protect the >>>> company's interests and to remove potential malware. Electronic messages >>>> may be intercepted, amended, lost or deleted, or contain viruses. >>>> >>>> >>>> >>>> Internal >>>> >>>>

1 0

Updating a CentOS7-Nautilus cluster to CentOS8-Pacific
by Massimo Sgaravatto 26 Apr '21

26 Apr '21

Hi I have a ceph cluster running Nautilus. The ceph services are hosted on CentOS7 servers. Right now I have: - 3 servers, each one running MON+MGR - 10 servers running OSDs - 2 servers running RGW I need to update this cluster to CentOS8 (actually CentOS stream 8) and Pacific. What is the update path that you would suggest ? I was thinking: CentOS7-Nautilus --> CentOS8-Nautilus --> CentOS8-Pacific but I don't know if this really the best solution. For example, as far as I understand ceph-deploy isn't available on CentOS8, and there is no documentation on how to manually (i.e without using ceph-deploy) re-deploy RGW on CentOS8 Thanks, Massimo

1 0

PG can't deep and simple scrub after unfound data delete
by Szabo, Istvan (Agoda) 26 Apr '21

26 Apr '21

Hi, I have a pg where it has been run the following command: ceph pg 44.1aa mark_unfound_lost delete After the cluster never report the unknown pgs which was actually the goal to ran this. However this pg is now inconsistent and can't be deepscrubbed. ceph health detail HEALTH_ERR 214275 scrub errors; Possible data damage: 1 pg inconsistent; 1 pgs not deep-scrubbed in time [ERR] OSD_SCRUB_ERRORS: 214275 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 44.1aa is active+clean+inconsistent, acting [59,128,127,43] [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100 ceph pg dump pgs_brief|grep 'ACTING_PRIMARY\|44.1aa' dumped pgs_brief PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 44.1aa active+clean+inconsistent [59,128,127,43] 59 [59,128,127,43] 59 Any idea what to do with it? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

RBD tuning for virtualization (all flash)
by by morphin 25 Apr '21

25 Apr '21

Hello. We're running 1000vm on 28 node with 6 ssd (no seperate db device) and these vms are Mostly win10. 2 lvm osd Per 4tb device total 288osd and one RBD pool with 8192 PG. Replication 3. Ceph version : nautilus 14.2.16 I'm looking for all flash RBD tuning. This is good test env and tomorrow gonna be prod. In vms I see 1800mb/s read and 900mb/s write (qemu = write back) With good tuning I believe this cluster will go further. Do you have any special suggestion? Have a nice weekend.

1 0

Pacific, one of monitor service doesnt response.
by Cem Zafer 24 Apr '21

24 Apr '21

Hi, I have no problem with my cluster before but when upgraded from 15.2.11 to 16.2.0 one of my ceph monitor service does not work. I have looked at the service log but cant figure it out. Can someone have any idea what could be wrong with that monitor service? Here is the service attached log. Regards root@ceph100:~# ceph orch ps NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID alertmanager.ceph100 ceph100 running (102m) 7m ago 3d - 0.20.0 0881eb8f169f b62ac302b319 crash.ceph100 ceph100 running (102m) 7m ago 3d - 16.2.0 24ecd6d5f14c e786dc48b429 crash.ceph101 ceph101 running (99m) 5m ago 3d - 16.2.0 24ecd6d5f14c a2cefd673a5f crash.ceph102 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c f9f8a337ba43 grafana.ceph100 ceph100 running (102m) 7m ago 3d - 6.7.4 80728b29ad3f 60381a67b6a2 mgr.ceph100.abnyjw ceph100 running (102m) 7m ago 3d *:8443 *:9283 16.2.1 c757e4a3636b 67f98b3ad44d mgr.ceph102.qqcrik ceph102 running (101m) 7m ago 8h *:8443 *:9283 16.2.1 c757e4a3636b 00a6eeb2a9eb mon.ceph100 ceph100 running (102m) 7m ago 3d - 16.2.0 24ecd6d5f14c 6bce0a137567 mon.ceph101 ceph101 stopped 5m ago 3d - <unknown> <unknown> <unknown> mon.ceph102 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c 9ca0e1c7ecd9 node-exporter.ceph100 ceph100 running (102m) 7m ago 3d - 0.18.1 e5a616e4b9cf b8ae7aa0fae1 node-exporter.ceph101 ceph101 running (99m) 5m ago 3d - 0.18.1 e5a616e4b9cf 945cdd081979 node-exporter.ceph102 ceph102 running (101m) 7m ago 3d - 0.18.1 e5a616e4b9cf d9a967358fbe osd.0 ceph100 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c 8f74ba97c27e osd.1 ceph100 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c fe0eed6a2cf4 osd.2 ceph101 running (99m) 5m ago 3d - 16.2.0 24ecd6d5f14c 1fd53036a139 osd.3 ceph101 running (99m) 5m ago 3d - 16.2.0 24ecd6d5f14c 329496f3a03e osd.4 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c cb16b76f0fc4 osd.5 ceph102 running (101m) 7m ago 3d - 16.2.0 24ecd6d5f14c c5d4e56812af prometheus.ceph100 ceph100 running (102m) 7m ago 3d - 2.18.1 de242295e225 b3f1c51d7136

2 1

Different ceph versions on nodes in cluster
by mabi 24 Apr '21

24 Apr '21

Hello, I was wondering if it is ok to have different Ceph version numbers among the nodes in a Ceph cluster? I am asking because I started setting up a new Octopus cluster with cephadm bootstrap which installed my initial node with version 15.2.10 and added a new mon node today and that new node got installed with 15.2.11... What is the best practice here? Best regards, Mabi

3 4

How to get rbd block device IOPS and BW performance showing?
by huxiaoyu＠horebdata.cn 24 Apr '21

24 Apr '21

Dear Ceph folks, I have a Luminous ceph cluster serving rbd block devices for an openstack cluster. My team leader would like me to show the performance view of these rbd block devices, including IOPS and BW, to get to know which rbd consumes most. I heard that Nautilous can do this , but how to get those statistics for Luminous version? Any help would be highly appreciated, thanks in advance, Samuel huxiaoyu(a)horebdata.cn

2 1

wrong socket path with ceph daemonperf
by Fabrice Bacchella 24 Apr '21

24 Apr '21

I doing a custom, manual installation of ceph. I changed the cluster name from the default one to 'ngceph'. I changed the installation path to /data/ceph. A lot of thing correctly resolves the settings. But one command fails: CLUSTER=/data/ceph/conf/ngceph.conf CLUSTER=ngceph ceph daemonperf client.radosgw.fa43 list ... Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ceph_daemon.py", line 73, in admin_socket b'{"prefix": "get_command_descriptions"}') File "/usr/lib/python3.6/site-packages/ceph_daemon.py", line 49, in do_sockio sock.connect(path) FileNotFoundError: [Errno 2] No such file or directory ... And indeed, after a few check, I found the culprit: CLUSTER=/data/ceph/conf/ngceph.conf CLUSTER=ngceph strace -f -e trace=file ceph daemonperf client.radosgw.fa43 list ... [pid 3431] execve("/bin/ceph-conf", ["ceph-conf", "--name", "client.radosgw.fa43", "--show-config-value", "admin_socket"], 0x5651c6e1cb30 /* 31 vars */) = 0 So I tried this command: CLUSTER=/data/ceph/conf/ngceph.conf CLUSTER=ngceph ceph-conf --name client.radosgw.fa43 --show-config-value admin_socket /var/run/ceph/ceph-client.radosgw.fa43.asok But the real path for the socket is: ls -l /var/run/ceph/ngceph-client.radosgw.fa43.2521.94787303273792.asok srwxr-xr-x 1 ceph ceph 0 Apr 22 21:47 /var/run/ceph/ngceph-client.radosgw.fa43.2521.94787303273792.asok It does not use the cluster name, and I wonder what is this 2521.94787303273792 coming from. Would it fails even with a standard cluster name ? The version I'm using is a rpm build from ceph: $ rpm -qf /bin/ceph-conf ceph-common-16.2.1-0.el8.x86_64

2 4

ceph osd replace SD card
by For 99 24 Apr '21

24 Apr '21

Hi every body. I have ceph cluster. One of OSD host os on sd card that have 6 osd hdd is crashed, so I need to change sd card. how can I reconnect 6 osd hard disks to new os without destroy the data? Thanks in advance.

3 2

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 23 Apr '21

23 Apr '21

Thank you so much~ I will follow your guide to repair index and let-sync the bucket. 😬 Will let you know the result after I do this. Thanks for your help again~ Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月23日周五下午10:19写道： > that my steps, it may be helpfull, > > if buckets objects not sync and bucket sync not work then fix index in > master and then run bucket sync > master:radosgw-admin bucket check -b bucket1 --fix --check-objects > secondary:radosgw-admin bucket sync run --source-zone us-west > --bucket=bucket1 > > > On Fri, Apr 23, 2021 at 7:07 PM 特木勒 <twl007(a)gmail.com> wrote: > >> I have not tried to fix bucket index. >> >> For error list I have checked and it looks good. >> >> The strange thing is only the objects that created after multiple site >> will be synced. And after sync recovered from networking issues, secondary >> cluster could pull loss data during networking downtime. So look like it’s >> working, just I do not know why only old objects will not be synced.🙁 >> >> Thanks for your suggestion and I will give it a try. >> >> Have a good day~ >> >> Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月23日周五下午9:00写道： >> >>> Maybe you can try to fix bucket Index from source zone or you have to >>> check is there any error in error list. >>> >>> On Fri, 23 Apr 2021 at 6:20 PM, 特木勒 <twl007(a)gmail.com> wrote: >>> >>>> Hi Istvan: >>>> >>>> We just upgraded whole cluster to 15.2.10 and the multiple site still >>>> cannot sync whole objects to secondary cluster. 🙁 >>>> >>>> Do you have any suggestions on this? And I open another issues in ceph >>>> tracker site: >>>> https://tracker.ceph.com/issues/50474 >>>> >>>> Hope someone could go to check this issue. >>>> >>>> Thanks >>>> >>>> 特木勒 <twl007(a)gmail.com>于2021年3月22日周一下午9:08写道： >>>> >>>> > Thank you~ >>>> > >>>> > I will try to upgrade cluster too. Seem like this is the only way for >>>> now. >>>> > 😭 >>>> > >>>> > I will let you know once I complete testing. :) >>>> > >>>> > Have a good day >>>> > >>>> > Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>于2021年3月22日周一下午3:38写道： >>>> > >>>> >> Yeah, doesn't work. Last week they fixed my problem ticket which >>>> caused >>>> >> the crashes, and due to the crashes stopped the replication I'll >>>> give a try >>>> >> this week again after the update if the daemon doesn't crash, maybe >>>> it will >>>> >> work, because if crash hasn't happened, the data was synced. Fingers >>>> >> crossed ;) Don't give up 😄 >>>> >> ------------------------------ >>>> >> *From:* 特木勒 <twl007(a)gmail.com> >>>> >> *Sent:* Monday, March 22, 2021 1:38 PM >>>> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> >> *Cc:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> >>>> >> >>>> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple >>>> Site >>>> >> does not sync olds data >>>> >> >>>> >> Hi Istvan: >>>> >> >>>> >> Do you have any update on directional sync? >>>> >> >>>> >> I am trying to upgrade cluster to 15.2.10 to see if the problem is >>>> >> solved. :( >>>> >> >>>> >> Thanks >>>> >> >>>> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年3月1日周一 >>>> 上午10:01写道： >>>> >> >>>> >> So-so. I had some interruption so it failed on one site, but the >>>> other is >>>> >> kind of working. This is the first time when I saw data caught up in >>>> the >>>> >> radosgw-admin data sync status on 1 side. >>>> >> >>>> >> Today will finish the other problematic site, I’ll let you know the >>>> >> result is it working or not. >>>> >> >>>> >> >>>> >> >>>> >> Istvan Szabo >>>> >> Senior Infrastructure Engineer >>>> >> --------------------------------------------------- >>>> >> Agoda Services Co., Ltd. >>>> >> e: istvan.szabo(a)agoda.com >>>> >> --------------------------------------------------- >>>> >> >>>> >> >>>> >> >>>> >> *From:* 特木勒 <twl007(a)gmail.com> >>>> >> *Sent:* Sunday, February 28, 2021 1:34 PM >>>> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >>>> >> *Cc:* ceph-users(a)ceph.io >>>> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple >>>> Site >>>> >> does not sync olds data >>>> >> >>>> >> >>>> >> >>>> >> Email received from outside the company. If in doubt don't click >>>> links >>>> >> nor open attachments! >>>> >> ------------------------------ >>> >>> >>>> >> >>>> >> Hi Istvan: >>>> >> >>>> >> >>>> >> >>>> >> Thanks for your reply. >>>> >> >>>> >> >>>> >> >>>> >> Does directional sync solve the problem? I tried to run >>>> `radosgw-admin >>>> >> sync init`, bit it still did not work. :( >>>> >> >>>> >> >>>> >> >>>> >> Thanks >>>> >> >>>> >> >>>> >> >>>> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年2月26日周五 >>>> 上午7:47写道： >>>> >> >>>> >> Same for me, 15.2.8 also. >>>> >> I’m trying directional sync now, looks like symmetrical has issue. >>>> >> >>>> >> Istvan Szabo >>>> >> Senior Infrastructure Engineer >>>> >> --------------------------------------------------- >>>> >> Agoda Services Co., Ltd. >>>> >> e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> >>>> >> --------------------------------------------------- >>>> >> >>>> >> On 2021. Feb 26., at 1:03, 特木勒 <twl007(a)gmail.com> wrote: >>>> >> >>>> >> Email received from outside the company. If in doubt don't click >>>> links >>>> >> nor open attachments! >>>> >> ________________________________ >>>> >> >>>> >> Hi all: >>>> >> >>>> >> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) >>>> >> >>>> >> I have a strange question, I just create a multiple site for Ceph >>>> cluster. >>>> >> But I notice the old data of source cluster is not synced. Only new >>>> data >>>> >> will be synced into second zone cluster. >>>> >> >>>> >> Is there anything I need to do to enable full sync for bucket or >>>> this is a >>>> >> bug? >>>> >> >>>> >> Thanks >>>> >> _______________________________________________ >>>> >> ceph-users mailing list -- ceph-users(a)ceph.io >>>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >> >>>> >> ________________________________ >>>> >> This message is confidential and is for the sole use of the intended >>>> >> recipient(s). It may also be privileged or otherwise protected by >>>> copyright >>>> >> or other legal rules. If you have received it by mistake please let >>>> us know >>>> >> by reply email and delete it from your system. It is prohibited to >>>> copy >>>> >> this message or disclose its content to anyone. Any confidentiality >>>> or >>>> >> privilege is not waived or lost by any mistaken delivery or >>>> unauthorized >>>> >> disclosure of the message. All messages sent to and from Agoda may be >>>> >> monitored to ensure compliance with company policies, to protect the >>>> >> company's interests and to remove potential malware. Electronic >>>> messages >>>> >> may be intercepted, amended, lost or deleted, or contain viruses. >>>> >> >>>> >> >>>> >> Internal >>>> >> >>>> > >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>

1 0

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 23 Apr '21

23 Apr '21

I have not tried to fix bucket index. For error list I have checked and it looks good. The strange thing is only the objects that created after multiple site will be synced. And after sync recovered from networking issues, secondary cluster could pull loss data during networking downtime. So look like it’s working, just I do not know why only old objects will not be synced.🙁 Thanks for your suggestion and I will give it a try. Have a good day~ Amit Ghadge <amitg.b14(a)gmail.com>于2021年4月23日周五下午9:00写道： > Maybe you can try to fix bucket Index from source zone or you have to > check is there any error in error list. > > On Fri, 23 Apr 2021 at 6:20 PM, 特木勒 <twl007(a)gmail.com> wrote: > >> Hi Istvan: >> >> We just upgraded whole cluster to 15.2.10 and the multiple site still >> cannot sync whole objects to secondary cluster. 🙁 >> >> Do you have any suggestions on this? And I open another issues in ceph >> tracker site: >> https://tracker.ceph.com/issues/50474 >> >> Hope someone could go to check this issue. >> >> Thanks >> >> 特木勒 <twl007(a)gmail.com>于2021年3月22日周一下午9:08写道： >> >> > Thank you~ >> > >> > I will try to upgrade cluster too. Seem like this is the only way for >> now. >> > 😭 >> > >> > I will let you know once I complete testing. :) >> > >> > Have a good day >> > >> > Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>于2021年3月22日周一下午3:38写道： >> > >> >> Yeah, doesn't work. Last week they fixed my problem ticket which caused >> >> the crashes, and due to the crashes stopped the replication I'll give >> a try >> >> this week again after the update if the daemon doesn't crash, maybe it >> will >> >> work, because if crash hasn't happened, the data was synced. Fingers >> >> crossed ;) Don't give up 😄 >> >> ------------------------------ >> >> *From:* 特木勒 <twl007(a)gmail.com> >> >> *Sent:* Monday, March 22, 2021 1:38 PM >> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >> >> *Cc:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> >> >> >> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >> >> does not sync olds data >> >> >> >> Hi Istvan: >> >> >> >> Do you have any update on directional sync? >> >> >> >> I am trying to upgrade cluster to 15.2.10 to see if the problem is >> >> solved. :( >> >> >> >> Thanks >> >> >> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年3月1日周一上午10:01写道： >> >> >> >> So-so. I had some interruption so it failed on one site, but the other >> is >> >> kind of working. This is the first time when I saw data caught up in >> the >> >> radosgw-admin data sync status on 1 side. >> >> >> >> Today will finish the other problematic site, I’ll let you know the >> >> result is it working or not. >> >> >> >> >> >> >> >> Istvan Szabo >> >> Senior Infrastructure Engineer >> >> --------------------------------------------------- >> >> Agoda Services Co., Ltd. >> >> e: istvan.szabo(a)agoda.com >> >> --------------------------------------------------- >> >> >> >> >> >> >> >> *From:* 特木勒 <twl007(a)gmail.com> >> >> *Sent:* Sunday, February 28, 2021 1:34 PM >> >> *To:* Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> >> >> *Cc:* ceph-users(a)ceph.io >> >> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site >> >> does not sync olds data >> >> >> >> >> >> >> >> Email received from outside the company. If in doubt don't click links >> >> nor open attachments! >> >> ------------------------------ > > >> >> >> >> Hi Istvan: >> >> >> >> >> >> >> >> Thanks for your reply. >> >> >> >> >> >> >> >> Does directional sync solve the problem? I tried to run `radosgw-admin >> >> sync init`, bit it still did not work. :( >> >> >> >> >> >> >> >> Thanks >> >> >> >> >> >> >> >> Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com> 于2021年2月26日周五上午7:47写道： >> >> >> >> Same for me, 15.2.8 also. >> >> I’m trying directional sync now, looks like symmetrical has issue. >> >> >> >> Istvan Szabo >> >> Senior Infrastructure Engineer >> >> --------------------------------------------------- >> >> Agoda Services Co., Ltd. >> >> e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> >> >> --------------------------------------------------- >> >> >> >> On 2021. Feb 26., at 1:03, 特木勒 <twl007(a)gmail.com> wrote: >> >> >> >> Email received from outside the company. If in doubt don't click links >> >> nor open attachments! >> >> ________________________________ >> >> >> >> Hi all: >> >> >> >> ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) >> >> >> >> I have a strange question, I just create a multiple site for Ceph >> cluster. >> >> But I notice the old data of source cluster is not synced. Only new >> data >> >> will be synced into second zone cluster. >> >> >> >> Is there anything I need to do to enable full sync for bucket or this >> is a >> >> bug? >> >> >> >> Thanks >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> ________________________________ >> >> This message is confidential and is for the sole use of the intended >> >> recipient(s). It may also be privileged or otherwise protected by >> copyright >> >> or other legal rules. If you have received it by mistake please let us >> know >> >> by reply email and delete it from your system. It is prohibited to copy >> >> this message or disclose its content to anyone. Any confidentiality or >> >> privilege is not waived or lost by any mistaken delivery or >> unauthorized >> >> disclosure of the message. All messages sent to and from Agoda may be >> >> monitored to ensure compliance with company policies, to protect the >> >> company's interests and to remove potential malware. Electronic >> messages >> >> may be intercepted, amended, lost or deleted, or contain viruses. >> >> >> >> >> >> Internal >> >> >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

1 0

certificates between msg and radosgw
by Fabrice Bacchella 23 Apr '21

23 Apr '21

I've just activated certificates on my radosgw, and the manager is unable to talk to it. On the manager logs, I see: 2021-04-23T15:05:03.917+0200 7f4109309700 0 [dashboard ERROR urllib3.connection] Certificate did not match expected hostname: XXX Certificate: {...)} 2021-04-23T15:05:03.918+0200 7f4109309700 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request self._validate_conn(conn) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn conn.connect() File "/lib/python3.6/site-packages/urllib3/connection.py", line 364, in connect _match_hostname(cert, self.assert_hostname or server_hostname) File "/lib/python3.6/site-packages/urllib3/connection.py", line 374, in _match_hostname match_hostname(cert, asserted_hostname) File "/lib64/python3.6/ssl.py", line 287, in match_hostname % (hostname, ', '.join(map(repr, dnsnames)))) ssl.CertificateError: hostname 'SOMEIP' doesn't match either of 'XXX' So the manager is communicating to the gateway using the IP. So I configured the names in ceph.conf: [client.radosgw.ID] rgw_data = /data/ceph/data/radosgw/$cluster.ID log_file = /data/ceph/logs/$cluster-radosgw.ID.log rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 ssl_certificate=/data/ceph/conf/ceph.crt ssl_private_key=/data/ceph/conf/ceph.key" rgw_dns_name = XXX But it's seems that the line rgw_dns_name is not used by the manager. Where does it resolve the name to use for the connection ?

1 0

Re: rbd snap create now working and just hangs forever
by Ilya Dryomov 23 Apr '21

23 Apr '21

On Fri, Apr 23, 2021 at 1:12 PM Boris Behrens <bb(a)kervyn.de> wrote: > > > > Am Fr., 23. Apr. 2021 um 13:00 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> On Fri, Apr 23, 2021 at 12:46 PM Boris Behrens <bb(a)kervyn.de> wrote: >> > >> > >> > >> > Am Fr., 23. Apr. 2021 um 12:16 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> On Fri, Apr 23, 2021 at 12:03 PM Boris Behrens <bb(a)kervyn.de> wrote: >> >> > >> >> > >> >> > >> >> > Am Fr., 23. Apr. 2021 um 11:52 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> >> >> >> >> >> This snippet confirms my suspicion. Unfortunately without a verbose >> >> >> log from that VM from three days ago (i.e. when it got into this state) >> >> >> it's hard to tell what exactly went wrong. >> >> >> >> >> >> The problem is that the VM doesn't consider itself to be the rightful >> >> >> owner of the lock and so when "rbd snap create" requests the lock from >> >> >> it in order to make a snapshot, the VM just ignores the request because >> >> >> even though it owns the lock, its record appears to be of sync. >> >> >> >> >> >> I'd suggest to kick it by restarting osd36. If the VM is active, it >> >> >> should reacquire the lock and hopefully update its internal record as >> >> >> expected. If "rbd snap create" still hangs after that, it would mean >> >> >> that we have a reproducer and can gather logs on the VM side. >> >> >> >> >> >> What version of qemu/librbd and ceph is in use (both on the VM side and >> >> >> on the side you are running "rbd snap create"? >> >> >> >> >> > I just stopped the OSD, waited some seconds and started it again. >> >> > I still can't create snapshots. >> >> > >> >> > Ceph version is 14.2.18 accross the board >> >> > qemu is 4.1.0-1 >> >> > as we use krbd, the kernel version is 5.2.9-arch1-1-ARCH >> >> > >> >> > How can I gather more logs to debug it? >> >> >> >> Are you saying that this image is mapped and the lock is held by the >> >> kernel client? It doesn't look that way from the logs you shared. >> > >> > We use krbd instead of librbd (at least this is what I think I know), but qemu is doing the kvm/rbd stuff. >> >> I'm going to assume that by "qemu is doing the kvm/rbd stuff", you >> mean that you are using the librbd driver inside qemu and that this >> image is opened by qemu (i.e. that driver). If you don't know what >> access method is being used, debugging this might be challenging ;) >> >> Let's start with the same output: "rbd lock ls", "rbd status" and "rbd >> snap create --debug-ms=1 --debug-rbd=20". It should be different after >> osd36 was restarted. > > Here is the new one: https://pastebin.com/6qTsJK6W > Ah ok, this CPU node still got the old thing and uses librbd to work with rbd instead of krbd. Sorry, I forgot that simply restarting the OSD doesn't trigger the code path that I'm hoping would cause librbd inside the VM to update its state. I took a look at the code and I think there are a couple of ways to do it (listed in the order of preference): - cut the network between the VM and the cluster for more than 30 seconds; it should be done externally so that to the VM it looks like a long network blip - stop the VM process for more than 30 seconds $ PID=<pid of qemu-system-x86_64 process> $ kill -STOP $PID && sleep 40 && kill -CONT $PID - stop the osd36 process for more than 30 seconds with "nodown" flag set $ ceph osd set nodown $ PID=<pid of osd36 process> $ kill -STOP $PID && sleep 40 && kill -CONT $PID $ ceph osd unset nodown Thanks, Ilya

1 0

Re: rbd snap create now working and just hangs forever
by Ilya Dryomov 23 Apr '21

23 Apr '21

On Fri, Apr 23, 2021 at 9:16 AM Boris Behrens <bb(a)kervyn.de> wrote: > > > > Am Do., 22. Apr. 2021 um 20:59 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> On Thu, Apr 22, 2021 at 7:33 PM Boris Behrens <bb(a)kervyn.de> wrote: >> > >> > >> > >> > Am Do., 22. Apr. 2021 um 18:30 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> On Thu, Apr 22, 2021 at 6:00 PM Boris Behrens <bb(a)kervyn.de> wrote: >> >> > >> >> > >> >> > >> >> > Am Do., 22. Apr. 2021 um 17:27 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> >> >> On Thu, Apr 22, 2021 at 5:08 PM Boris Behrens <bb(a)kervyn.de> wrote: >> >> >> > >> >> >> > >> >> >> > >> >> >> > Am Do., 22. Apr. 2021 um 16:43 Uhr schrieb Ilya Dryomov <idryomov(a)gmail.com>: >> >> >> >> >> >> >> >> On Thu, Apr 22, 2021 at 4:20 PM Boris Behrens <bb(a)kervyn.de> wrote: >> >> >> >> > >> >> >> >> > Hi, >> >> >> >> > >> >> >> >> > I have a customer VM that is running fine, but I can not make snapshots >> >> >> >> > anymore. >> >> >> >> > rbd snap create rbd/IMAGE@test-bb-1 >> >> >> >> > just hangs forever. >> >> >> >> >> >> >> >> Hi Boris, >> >> >> >> >> >> >> >> Run >> >> >> >> >> >> >> >> $ rbd snap create rbd/IMAGE@test-bb-1 --debug-ms=1 --debug-rbd=20 >> >> >> >> >> >> >> >> let it hang for a few minutes and attach the output. >> >> >> > >> >> >> > >> >> >> > I just pasted a short snip here: https://pastebin.com/B3Xgpbzd >> >> >> > If you need more I can give it to you, but the output is very large. >> >> >> >> >> >> Paste the first couple thousand lines (i.e. from the very beginning), >> >> >> that should be enough. >> >> >> >> >> > sure: https://pastebin.com/GsKpLbqG >> >> > >> >> > good luck :) >> >> >> >> What is the output of "rbd status"? I know you said it shows one >> >> watcher, but I need to see it. >> >> >> >> >> > sure >> > # rbd status rbd/IMAGE >> > Watchers: >> > watcher=[fd00:2380:2:43::11]:0/3919389201 client.136378749 cookie=139968010125312 >> > > Hi Ilya, > thank you a lot for your support. > > This might be other hanging snapshot sheduler that got removed afterwards. > Sorry for that. > > https://pastebin.com/TBZs7Mvb > > I just created a new paste and added status and lock ls at the top and at the bottom. > The 2nd watcher disaperas after a minute or so. > All commands are done within one minute. This snippet confirms my suspicion. Unfortunately without a verbose log from that VM from three days ago (i.e. when it got into this state) it's hard to tell what exactly went wrong. The problem is that the VM doesn't consider itself to be the rightful owner of the lock and so when "rbd snap create" requests the lock from it in order to make a snapshot, the VM just ignores the request because even though it owns the lock, its record appears to be of sync. I'd suggest to kick it by restarting osd36. If the VM is active, it should reacquire the lock and hopefully update its internal record as expected. If "rbd snap create" still hangs after that, it would mean that we have a reproducer and can gather logs on the VM side. What version of qemu/librbd and ceph is in use (both on the VM side and on the side you are running "rbd snap create"? Thanks, Ilya

2 4

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 23 Apr '21

23 Apr '21

On Fri, Apr 23, 2021 at 6:57 AM Cem Zafer <cemzafer(a)gmail.com> wrote: > > Hi Ilya, > Sorry, totally my mistake. I just saw that the configuration on mars like that. > > auth_cluster_required = none > auth_service_required = none > auth_client_required = none > > So I changed none to cephx, solved the problem. > Thanks for your patience and support. Yup, that would do it ;) Thanks, Ilya

1 0

cephadm and VRF
by Karsten Nielsen 23 Apr '21

23 Apr '21

Hi cephers, I am a happy user of ceph and cephadm, we run it in a hyberconverged setup. We are moving into running bgp on to the node and have ECMP. My networking team is asking to run VRF on the nodes we will have a VRF for client traffic and a VRF for the storage network. Is that supported and will the daemons bind the right interfaces when specifying the client and cluster network ? Thanks, - Karsten

1 0

Re: s3 requires twice the space it should use
by Boris Behrens 23 Apr '21

23 Apr '21

Ah you are right. [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size_hdd { "bluestore_min_alloc_size_hdd": "65536" } But I also checked how many objects our s3 hold and the numbers just do not add up. There are only 26509200 objects, which would result in around 1TB "waste" if every object would be empty. I think the problem began when I updated the PG count from 1024 to 2048. Could there be an issue where the data is written twice? Am Do., 15. Apr. 2021 um 16:48 Uhr schrieb Amit Ghadge <amitg.b14(a)gmail.com >: > verify those two parameter values ,bluestore_min_alloc_size_hdd & > bluestore_min_alloc_size_sdd, If you are using hdd disk then > bluestore_min_alloc_size_hdd are applicable. > > On Thu, Apr 15, 2021 at 8:06 PM Boris Behrens <bb(a)kervyn.de> wrote: > >> So, I need to live with it? A value of zero leads to use the default? >> [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size >> { >> "bluestore_min_alloc_size": "0" >> } >> >> I also checked the fragmentation on the bluestore OSDs and it is around >> 0.80 - 0.89 on most OSDs. yikes. >> [root@s3db1 ~]# ceph daemon osd.23 bluestore allocator score block >> { >> "fragmentation_rating": 0.85906054329923576 >> } >> >> The problem I currently have is, that I barely keep up with adding OSD >> disks. >> >> Am Do., 15. Apr. 2021 um 16:18 Uhr schrieb Amit Ghadge < >> amitg.b14(a)gmail.com>: >> >>> size_kb_actual are actually bucket object size but on OSD level the >>> bluestore_min_alloc_size default 64KB and SSD are 16KB >>> >>> >>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/a… >>> >>> -AmitG >>> >>> On Thu, Apr 15, 2021 at 7:29 PM Boris Behrens <bb(a)kervyn.de> wrote: >>> >>>> Hi, >>>> >>>> maybe it is just a problem in my understanding, but it looks like our s3 >>>> requires twice the space it should use. >>>> >>>> I ran "radosgw-admin bucket stats", and added all "size_kb_actual" >>>> values >>>> up and divided to TB (/1024/1024/1024). >>>> The resulting space is 135,1636733 TB. When I tripple it because of >>>> replication I end up with around 405TB which is nearly half the space of >>>> what ceph df tells me. >>>> >>>> Hope someone can help me. >>>> >>>> ceph df shows >>>> RAW STORAGE: >>>> CLASS SIZE AVAIL USED RAW USED %RAW >>>> USED >>>> hdd 1009 TiB 189 TiB 820 TiB 820 TiB >>>> 81.26 >>>> TOTAL 1009 TiB 189 TiB 820 TiB 820 TiB >>>> 81.26 >>>> >>>> POOLS: >>>> POOL ID PGS STORED >>>> OBJECTS >>>> USED %USED MAX AVAIL >>>> rbd 0 64 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> .rgw.root 1 64 99 KiB >>>> 119 >>>> 99 KiB 0 18 TiB >>>> eu-central-1.rgw.control 2 64 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> eu-central-1.rgw.data.root 3 64 1.0 MiB >>>> 3.15k >>>> 1.0 MiB 0 18 TiB >>>> eu-central-1.rgw.gc 4 64 71 MiB >>>> 32 >>>> 71 MiB 0 18 TiB >>>> eu-central-1.rgw.log 5 64 267 MiB >>>> 564 >>>> 267 MiB 0 18 TiB >>>> eu-central-1.rgw.users.uid 6 64 2.8 MiB >>>> 6.91k >>>> 2.8 MiB 0 18 TiB >>>> eu-central-1.rgw.users.keys 7 64 263 KiB >>>> 6.73k >>>> 263 KiB 0 18 TiB >>>> eu-central-1.rgw.meta 8 64 384 KiB >>>> 1k >>>> 384 KiB 0 18 TiB >>>> eu-central-1.rgw.users.email 9 64 40 B >>>> 1 >>>> 40 B 0 18 TiB >>>> eu-central-1.rgw.buckets.index 10 64 10 GiB >>>> 67.61k >>>> 10 GiB 0.02 18 TiB >>>> eu-central-1.rgw.buckets.data 11 2048 264 TiB >>>> 138.31M >>>> 264 TiB 83.37 18 TiB >>>> eu-central-1.rgw.buckets.non-ec 12 64 297 MiB >>>> 11.32k >>>> 297 MiB 0 18 TiB >>>> eu-central-1.rgw.usage 13 64 536 MiB >>>> 32 >>>> 536 MiB 0 18 TiB >>>> eu-msg-1.rgw.control 56 64 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> eu-msg-1.rgw.data.root 57 64 72 KiB >>>> 227 >>>> 72 KiB 0 18 TiB >>>> eu-msg-1.rgw.gc 58 64 300 KiB >>>> 32 >>>> 300 KiB 0 18 TiB >>>> eu-msg-1.rgw.log 59 64 835 KiB >>>> 242 >>>> 835 KiB 0 18 TiB >>>> eu-msg-1.rgw.users.uid 60 64 56 KiB >>>> 104 >>>> 56 KiB 0 18 TiB >>>> eu-msg-1.rgw.usage 61 64 37 MiB >>>> 25 >>>> 37 MiB 0 18 TiB >>>> eu-msg-1.rgw.users.keys 62 64 3.8 KiB >>>> 97 >>>> 3.8 KiB 0 18 TiB >>>> eu-msg-1.rgw.meta 63 64 607 KiB >>>> 1.60k >>>> 607 KiB 0 18 TiB >>>> eu-msg-1.rgw.buckets.index 64 64 71 MiB >>>> 119 >>>> 71 MiB 0 18 TiB >>>> eu-msg-1.rgw.users.email 65 64 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> eu-msg-1.rgw.buckets.data 66 64 2.9 TiB >>>> 1.16M >>>> 2.9 TiB 5.30 18 TiB >>>> eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB >>>> 354 >>>> 2.2 MiB 0 18 TiB >>>> default.rgw.control 69 32 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> default.rgw.data.root 70 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> default.rgw.gc 71 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> default.rgw.log 72 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> default.rgw.users.uid 73 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> fra-1.rgw.control 74 32 0 B >>>> 8 >>>> 0 B 0 18 TiB >>>> fra-1.rgw.meta 75 32 0 B >>>> 0 >>>> 0 B 0 18 TiB >>>> fra-1.rgw.log 76 32 50 B >>>> 28 >>>> 50 B 0 18 TiB >>>> >>>> >>>> -- >>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >>>> groÃƒ¼en Saal. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> groÃƒ¼en Saal. >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 4

cephadm: how to create more than 1 rgw per host
by ivan＠z1storage.com 23 Apr '21

23 Apr '21

Hello, According to the documentation, there's count-per-host key to 'ceph orch', but it does not work for me: :~# ceph orch apply rgw z1 sa-1 --placement='label:rgw count-per-host:2' --port=8000 --dry-run Error EINVAL: Host and label are mutually exclusive Why it says anything about Host if I don't specify any hosts, just labels? ~# ceph orch host ls HOST ADDR LABELS STATUS s101 s101 mon rgw s102 s102 mgr mon rgw s103 s103 mon rgw s104 s104 mgr mon rgw s105 s105 mgr mon rgw s106 s106 mon rgw s107 s107 mon rgw

4 7

Zero Reclaim/Trim on RBD image
by Vikas Rana 23 Apr '21

23 Apr '21

Hi Friends, We have multiple RBD images/devices. We used fstrim to reclaim space on XFS filesystem on RBD images and it works great. Some of the RBD images, we share as block devices to Vmware using SCST. Any suggestion on how to reclaim space on these devices? The images are about 20TB but showing as 100TB used inside ceph pool. Thanks, -Vikas

2 1

Metrics for object sizes
by Szabo, Istvan (Agoda) 23 Apr '21

23 Apr '21

Hi, Is there any clusterwise metric regarding object sizes? I'd like to collect some information about the users what is the object sizes in their buckets. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 22 Apr '21

22 Apr '21

On Thu, Apr 22, 2021 at 10:16 PM Cem Zafer <cemzafer(a)gmail.com> wrote: > > This client ceph-common version is 16.2.0, here are the outputs. > > indiana@mars:~$ ceph -v > ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable) > > indiana@mars:~$ dpkg -l | grep -i ceph-common > ii ceph-common 16.2.0-1focal amd64 common utilities to mount and interact with a ceph storage cluster > ii python3-ceph-common 16.2.0-1focal all Python 3 utility libraries for Ceph Is the problem local to "mars"? Are you able to "ceph -s" with 16.2.0 client from other hosts? I suspect there is something wrong with ceph.conf on mars. The error message is different: handle_auth_bad_method server allowed_methods [2] but i only support [1] instead of handle_auth_bad_method server allowed_methods [2] but i only support [2] Please run "ceph -s --debug_ms=1 --debug_monc=25 --debug_auth=25" and attach ceph.conf from mars. Thanks, Ilya

1 0

Configuring an S3 gateway
by Fabrice Bacchella 22 Apr '21

22 Apr '21

I'm trying to configure an S3 gateway with pacific and can't wrap my mind around. In the configuration file, my configuration is: [client.radosgw.fa41] rgw_data = /data/ceph/data/radosgw/$cluster.$id log_file = /data/ceph/logs/$cluster-radosgw.$id.log rgw_frontends = "beast ssl_endpoint=0.0.0.0:443 ssl_certificate=/data/ceph/conf/ceph.crt ssl_private_key=/data/ceph/conf/ceph.key" and radosgw ignore it: # /usr/bin/radosgw -d --cluster ngceph --name client.fa41 --setuser ceph --setgroup ceph 2021-04-22T20:19:44.362+0200 7fdf51416480 0 deferred set uid:gid to 167:167 (ceph:ceph) 2021-04-22T20:19:44.363+0200 7fdf51416480 0 ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable), process radosgw, pid 9780 2021-04-22T20:19:44.363+0200 7fdf51416480 0 framework: beast 2021-04-22T20:19:44.363+0200 7fdf51416480 0 framework conf key: port, val: 7480 2021-04-22T20:19:44.363+0200 7fdf51416480 1 radosgw_Main not setting numa affinity 2021-04-22T20:19:45.585+0200 7fdf51416480 0 framework: beast 2021-04-22T20:19:45.586+0200 7fdf51416480 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt 2021-04-22T20:19:45.586+0200 7fdf51416480 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key 2021-04-22T20:19:45.586+0200 7fdf51416480 0 starting handler: beast 2021-04-22T20:19:45.592+0200 7fdf51416480 0 WARNING: cannot open socket for endpoint=[::]:7480, Address family not supported by protocol 2021-04-22T20:19:45.627+0200 7fdf51416480 0 set uid:gid to 167:167 (ceph:ceph) 2021-04-22T20:19:45.811+0200 7fdf51416480 1 mgrc service_daemon_register rgw.134130 metadata {arch=x86_64,ceph_release=pacific,ceph_version=ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable),ceph_version_short=16.2.1,cpu=Intel Core Processor (Haswell, no TSX),distro=centos,distro_description=CentOS Linux 8,distro_version=8,frontend_config#0=beast port=7480,frontend_type#0=beast,hostname=fa41,id=fa41,kernel_description=#1 SMP Thu Apr 8 19:01:30 UTC 2021,kernel_version=4.18.0-240.22.1.el8_3.x86_64,mem_swap_kb=0,mem_total_kb=16211232,num_handles=1,os=Linux,pid=9780,zone_id=2dc75a54-8c59-42bc-98a8-35542fdc4e52,zone_name=default,zonegroup_id=d11b8d14-7608-4b1d-a548-09b5dd813a7a,zonegroup_name=default} I don't get what I'm missing. Is there any typo in the configuration that I'm missing ? I've verified using strace, and it reads the expected configuration file.

3 3

/ceph-osd-prestart.sh does use the configuration
by Fabrice Bacchella 22 Apr '21

22 Apr '21

I'm using a rpm build directly from ceph, ceph-osd-16.2.1-0.el8.x86_64 I tweaked the configuration with a custom cluster name and custom installation dir In the unit, there is the configuration: ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i And this script check the osd path with an hard coded value: data="/var/lib/ceph/osd/${cluster:-ceph}-$id" It doesn't use the key from the configuration file. But using ceph-conf would have solve the problem: $ CEPH_CONF=/data/ceph/conf/ngceph.conf CLUSTER=ngceph ceph-conf --name osd.1 --show-config-value osd_data /data/ceph/data/osd/ceph.1

1 0

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 22 Apr '21

22 Apr '21

On Thu, Apr 22, 2021 at 9:24 PM Cem Zafer <cemzafer(a)gmail.com> wrote: > > Sorry to disturb you again but changing the value to yes doesnt affect anything. Executing simple ceph command from the client replies the following error, again. I'm not so sure it is related with that parameter. > Have you any idea what could cause the problem? > > indiana@mars:~$ ceph -s > 2021-04-22T22:19:51.305+0300 7f20ea249700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] > [errno 13] RADOS permission denied (error connecting to the cluster) This looks like a different host/client. What version of ceph-common is installed on mars (or just run "ceph -v" instead of "ceph -s")? Thanks, Ilya

1 0

RGW objects has same marker and bucket id in different buckets.
by by morphin 22 Apr '21

22 Apr '21

Hello. I have a rgw s3 user and the user have 2 bucket. I tried to copy objects from old.bucket to new.bucket with rclone. (in the rgw client server) After I checked the object with "radosgw-admin --bucket=new.bucket object stat $i" and I saw old.bucket id and marker id also old bucket name in the object stats. Is rgw doing this for deduplication or is it a bug? If it's not a bug then If I delete the old bucket what will happen to these objects???

2 4

rbd snap create now working and just hangs forever
by Boris Behrens 22 Apr '21

22 Apr '21

Hi, I have a customer VM that is running fine, but I can not make snapshots anymore. rbd snap create rbd/IMAGE@test-bb-1 just hangs forever. When I checked the status with rbd status rbd/IMAGE it shows one watcher, the cpu node where the VM is running. What can I do to investigate further, without restarting the VM. This is the only affected VM and it stopped working three days ago.

2 5

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 22 Apr '21

22 Apr '21

On Thu, Apr 22, 2021 at 6:01 PM Cem Zafer <cemzafer(a)gmail.com> wrote: > > Thanks Ilya, pointing me out to the right direction. > So if I change the auth_allow_insecure_global_id_reclaim to true means older userspace clients allowed to the cluster? Yes, but upgrading all clients and setting it to false is recommended. Thanks, Ilya

1 0

Re: monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Ilya Dryomov 22 Apr '21

22 Apr '21

On Thu, Apr 22, 2021 at 5:04 PM Cem Zafer <cemzafer(a)gmail.com> wrote: > > Hi Ilya, > Yes you are correct, I have set auth_allow_insecure_global_id_reclaim to false. > Host ceph-common package version is 16.2.0 and the cluster ceph -v output is as follows. > > root@ceph100:~# ceph -v > ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) > Regards. Right, so because you set auth_allow_insecure_global_id_reclaim to false, older userspace clients, in this case 16.2.0, are not allowed to connect because they won't reclaim their global_id in a secure fashion. See https://docs.ceph.com/en/latest/security/CVE-2021-20288/ for details. Thanks, Ilya > > On Thu, Apr 22, 2021 at 4:49 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: >> >> On Thu, Apr 22, 2021 at 3:24 PM Cem Zafer <cemzafer(a)gmail.com> wrote: >> > >> > Hi, >> > I have recently add a new host to ceph and copied /etc/ceph directory to >> > the new host. When I execute the simple ceph command as "ceph -s", get the >> > following error. >> > >> > 021-04-22T14:50:46.226+0300 7ff541141700 -1 monclient(hunting): >> > handle_auth_bad_method server allowed_methods [2] but i only support [2] >> > 2021-04-22T14:50:46.226+0300 7ff540940700 -1 monclient(hunting): >> > handle_auth_bad_method server allowed_methods [2] but i only support [2] >> > 2021-04-22T14:50:46.226+0300 7ff533fff700 -1 monclient(hunting): >> > handle_auth_bad_method server allowed_methods [2] but i only support [2] >> > [errno 13] RADOS permission denied (error connecting to the cluster) >> > >> > When I looked at the syslog on the ceph cluster node, I saw that message >> > too. >> > >> > Apr 22 14:51:40 ceph100 bash[27979]: debug 2021-04-22T11:51:40.684+0000 >> > 7fe4d28cb700 0 cephx server client.admin: attempt to reclaim global_id >> > 264198 without presenting ticket >> > Apr 22 14:51:40 ceph100 bash[27979]: debug 2021-04-22T11:51:40.684+0000 >> > 7fe4d28cb700 0 cephx server client.admin: could not verify old ticket >> > >> > Anyone can help me out or assist to the right direction or link? >> >> Hi Cem, >> >> I take it that you upgraded to one of 14.2.20, 15.2.11 or 16.2.1 >> releases and then set auth_allow_insecure_global_id_reclaim to false? >> >> What version of ceph-common package is installed on that host? What is >> the output of "ceph -v"? >> >> Thanks, >> >> Ilya

1 0

monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
by Cem Zafer 22 Apr '21

22 Apr '21

Hi, I have recently add a new host to ceph and copied /etc/ceph directory to the new host. When I execute the simple ceph command as "ceph -s", get the following error. 021-04-22T14:50:46.226+0300 7ff541141700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2021-04-22T14:50:46.226+0300 7ff540940700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2021-04-22T14:50:46.226+0300 7ff533fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] [errno 13] RADOS permission denied (error connecting to the cluster) When I looked at the syslog on the ceph cluster node, I saw that message too. Apr 22 14:51:40 ceph100 bash[27979]: debug 2021-04-22T11:51:40.684+0000 7fe4d28cb700 0 cephx server client.admin: attempt to reclaim global_id 264198 without presenting ticket Apr 22 14:51:40 ceph100 bash[27979]: debug 2021-04-22T11:51:40.684+0000 7fe4d28cb700 0 cephx server client.admin: could not verify old ticket Anyone can help me out or assist to the right direction or link? Regards.

2 1

Download-Mirror eu.ceph.com misses Debian Release file
by Robert Sander 22 Apr '21

22 Apr '21

Hi, to whomever it may concern: The mirror server eu.ceph.com does to carry the Release files for 15.2.11 in https://eu.ceph.com/debian-15.2.11/dists/*/ and 16.2.1 in https://eu.ceph.com/debian-16.2.1/dists/*/ Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

After upgrade to 15.2.11 no access to cluster any more
by Robert Sander 22 Apr '21

22 Apr '21

Hi, I upgraded a test cluster to 15.2.11 and after the upgrade has been finished the cluster status was in HEALTH_WARN because of: AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED: mons are allowing insecure global_id reclaim According to the documentation at https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insec… I ran ceph config set mon auth_allow_insecure_global_id_reclaim false because no client was reporting using insecure global_ids. Now "ceph -s" cannot connect to the cluster any more: # ceph -s [errno 13] RADOS permission denied (error connecting to the cluster) What should I do? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2

_delete_some new onodes has appeared since PG removal started
by Dan van der Ster 21 Apr '21

21 Apr '21

Hi Igor, After updating to 14.2.19 and then moving some PGs around we have a few warnings related to the new efficient PG removal code, e.g. [1]. Is that something to worry about? Best Regards, Dan [1] /var/log/ceph/ceph-osd.792.log:2021-04-14 20:34:34.353 7fb2439d4700 0 osd.792 pg_epoch: 40906 pg[10.14b2s0( v 40734'290069 (33782'287000,40734'290069] lb MIN (bitwise) local-lis/les=33990/33991 n=36272 ec=4951/4937 lis/c 33990/33716 les/c/f 33991/33747/0 40813/40813/37166) [933,626,260,804,503,491]p933(0) r=-1 lpr=40813 DELETING pi=[33716,40813)/4 crt=40734'290069 unknown NOTIFY mbc={}] _delete_some additional unexpected onode list (new onodes has appeared since PG removal started[0#10:4d280000::::head#] /var/log/ceph/ceph-osd.851.log:2021-04-14 18:40:13.312 7fd87bded700 0 osd.851 pg_epoch: 40671 pg[10.133fs5( v 40662'288967 (33782'285900,40662'288967] lb MIN (bitwise) local-lis/les=33786/33787 n=13 ec=4947/4937 lis/c 40498/33714 les/c/f 40499/33747/0 40670/40670/33432) [859,199,913,329,439,79]p859(0) r=-1 lpr=40670 DELETING pi=[33714,40670)/4 crt=40662'288967 unknown NOTIFY mbc={}] _delete_some additional unexpected onode list (new onodes has appeared since PG removal started[5#10:fcc80000::::head#] /var/log/ceph/ceph-osd.851.log:2021-04-14 20:58:14.393 7fd87adeb700 0 osd.851 pg_epoch: 40906 pg[10.2e8s3( v 40610'288991 (33782'285900,40610'288991] lb MIN (bitwise) local-lis/les=33786/33787 n=161220 ec=4937/4937 lis/c 39826/33716 les/c/f 39827/33747/0 40617/40617/39225) [717,933,727,792,607,129]p717(0) r=-1 lpr=40617 DELETING pi=[33716,40617)/3 crt=40610'288991 unknown NOTIFY mbc={}] _delete_some additional unexpected onode list (new onodes has appeared since PG removal started[3#10:17400000::::head#] /var/log/ceph/ceph-osd.883.log:2021-04-14 18:55:16.822 7f78c485d700 0 osd.883 pg_epoch: 40857 pg[7.d4( v 40804'9911289 (35835'9908201,40804'9911289] lb MIN (bitwise) local-lis/les=40782/40783 n=195 ec=2063/1989 lis/c 40782/40782 les/c/f 40783/40844/0 40781/40845/40845) [877,870,894] r=-1 lpr=40845 DELETING pi=[40782,40845)/1 crt=40804'9911289 lcod 40804'9911288 unknown NOTIFY mbc={}] _delete_some additional unexpected onode list (new onodes has appeared since PG removal started[#7:2b000000::::head#]

4 13

EC Backfill Observations
by Josh Baergen 21 Apr '21

21 Apr '21

Hey all, I wanted to confirm my understanding of some of the mechanics of backfill in EC pools. I've yet to find a document that outlines this in detail; if there is one, please send it my way. :) Some of what I write below is likely in the "well, duh" category, but I tended towards completeness. First off, I understand that backfill reservations work the same way between replicated pools and EC pools. A local reservation is taken on the primary OSD, then a remote reservation on the backfill target(s), before the backfill is allowed to begin. Until this point, the backfill is in the backfill_wait state. When the backfill begins, though, is when the differences begin. Let's say we have an EC 3:2 PG that's backfilling from OSD 2 to OSD 5 (formatted here like pgs_brief): 1.1 active+remapped+backfilling [0,1,5,3,4] 0 [0,1,2,3,4] 0 The question in my mind was: Where is the data for this backfill coming from? In replicated pools, all reads come from the primary. However, in this case, the primary does not have the data in question; the primary has to either read the EC chunk from OSD 2, or it has to reconstruct it by reading from 3 of the OSDs in the acting set. Based on observation, I _think_ this is what happens: 1. As long as the PG is not degraded, the backfill read is simply forwarded by the primary to OSD 2. 2. Once the PG becomes degraded, the backfill read needs to use the reconstructing path, and begins reading from 3 of the OSDs in the acting set. Questions: 1. Can anyone confirm or correct my description of how EC backfill operates? In particular, in case 2 above, does it matter whether OSD 2 is the cause of degradation, for example? Does the read still get forwarded to a single OSD when it's parity chunks that are being moved via backfill? 2. I'm curious as to why a 3rd reservation, for the source OSD, wasn't introduced as a part of EC in Ceph. We've occasionally seen an OSD become overloaded because several backfills were reading from it simultaneously, and there's no way to control this via the normal osd_max_backfills mechanism. Is anyone aware of discussions to this effect? Thanks! Josh

2 4

MDS_TRIM 1 MDSs behind on trimming and
by Flemming Frandsen 21 Apr '21

21 Apr '21

I've just spent a couple of hours waiting for an MDS server to replay a journal that it was behind on and it seems to be getting worse. The system is not terribly busy, but there are 14 ops in flight that are very old and do not seem to go away on their own. Is there anything I can do to unwedge the mds server? root@dalmore:~# ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mds.dalmore(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 4107 secs MDS_SLOW_REQUEST 1 MDSs report slow requests mds.dalmore(mds.0): 14 slow requests are blocked > 30 secs MDS_TRIM 1 MDSs behind on trimming mds.dalmore(mds.0): Behind on trimming (3443/128) max_segments: 128, num_segments: 3443 root@dalmore:~# ceph daemon mds.dalmore dump_ops_in_flight { "ops": [ { "description": "client_request(client.9801215:348401522 readdir #0x2001fba1c83 2021-04-21 15:46:30.820531 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:46:30.822084", "age": 4020.3439449050002, "duration": 4020.343998116, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348401522", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348401522 }, "events": [ { "time": "2021-04-21 15:46:30.822084", "event": "initiated" }, { "time": "2021-04-21 15:46:30.822084", "event": "header_read" }, { "time": "2021-04-21 15:46:30.822090", "event": "throttled" }, { "time": "2021-04-21 15:46:30.822108", "event": "all_read" }, { "time": "2021-04-21 15:46:30.966336", "event": "dispatched" }, { "time": "2021-04-21 15:46:30.966419", "event": "acquired locks" }, { "time": "2021-04-21 16:01:40.084378", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084438", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9801215:348454506 setfilelock rule 1, type 4, owner 14336643358275911908, pid 32085, start 0, length 0, wait 0 #0x1001e694623 20 21-04-21 15:55:50.829175 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:55:50.832432", "age": 3460.3335972099999, "duration": 3460.333751787, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348454506", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348454506 }, "events": [ { "time": "2021-04-21 15:55:50.832432", "event": "initiated" }, { "time": "2021-04-21 15:55:50.832432", "event": "header_read" }, { "time": "2021-04-21 15:55:50.832439", "event": "throttled" }, { "time": "2021-04-21 15:55:50.832465", "event": "all_read" }, { "time": "2021-04-21 15:55:50.832608", "event": "dispatched" }, { "time": "2021-04-21 16:01:40.084448", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084451", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9143865:2383945773 getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:35:30.075110 caller_uid=0, caller_gid=0{})", "initiated_at": "2021-04-21 16:35:30.077995", "age": 1081.0880337159999, "duration": 1081.0882681850001, "type_data": { "flag_point": "failed to authpin, subtree is being exported", "reqid": "client.9143865:2383945773", "op_type": "client_request", "client_info": { "client": "client.9143865", "tid": 2383945773 }, "events": [ { "time": "2021-04-21 16:35:30.077995", "event": "initiated" }, { "time": "2021-04-21 16:35:30.077995", "event": "header_read" }, { "time": "2021-04-21 16:35:30.077999", "event": "throttled" }, { "time": "2021-04-21 16:35:30.078018", "event": "all_read" }, { "time": "2021-04-21 16:35:30.078091", "event": "dispatched" }, { "time": "2021-04-21 16:35:30.078156", "event": "failed to authpin, subtree is being exported" } ] } }, { "description": "client_request(client.9801215:348417261 lookup #0x10004cd0675/tasks 2021-04-21 15:49:09.248696 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:49:09.252834", "age": 3861.9131950870001, "duration": 3861.9135065690002, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348417261", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348417261 }, "events": [ { "time": "2021-04-21 15:49:09.252834", "event": "initiated" }, { "time": "2021-04-21 15:49:09.252834", "event": "header_read" }, { "time": "2021-04-21 15:49:09.252839", "event": "throttled" }, { "time": "2021-04-21 15:49:09.252856", "event": "all_read" }, { "time": "2021-04-21 15:49:09.252982", "event": "dispatched" }, { "time": "2021-04-21 15:49:09.253070", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:01:40.084439", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084444", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9143865:2383945793 getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:40:33.868658 caller_uid=0, caller_gid=0{})", "initiated_at": "2021-04-21 16:40:33.871565", "age": 777.29446343799998, "duration": 777.29486169500001, "type_data": { "flag_point": "failed to authpin, subtree is being exported", "reqid": "client.9143865:2383945793", "op_type": "client_request", "client_info": { "client": "client.9143865", "tid": 2383945793 }, "events": [ { "time": "2021-04-21 16:40:33.871565", "event": "initiated" }, { "time": "2021-04-21 16:40:33.871565", "event": "header_read" }, { "time": "2021-04-21 16:40:33.871572", "event": "throttled" }, { "time": "2021-04-21 16:40:33.871617", "event": "all_read" }, { "time": "2021-04-21 16:40:33.871699", "event": "dispatched" }, { "time": "2021-04-21 16:40:33.871909", "event": "failed to authpin, subtree is being exported" } ] } }, { "description": "client_request(client.9801215:348424659 lookup #0x10004cd0675/slaves 2021-04-21 15:50:01.592755 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:50:01.595195", "age": 3809.5708340870001, "duration": 3809.5713271260001, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348424659", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348424659 }, "events": [ { "time": "2021-04-21 15:50:01.595195", "event": "initiated" }, { "time": "2021-04-21 15:50:01.595195", "event": "header_read" }, { "time": "2021-04-21 15:50:01.595198", "event": "throttled" }, { "time": "2021-04-21 15:50:01.595210", "event": "all_read" }, { "time": "2021-04-21 15:50:01.595307", "event": "dispatched" }, { "time": "2021-04-21 15:50:01.595389", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:01:40.084444", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084448", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9801215:348456034 setfilelock rule 1, type 4, owner 14336643483181153700, pid 32085, start 0, length 0, wait 0 #0x1001e694623 20 21-04-21 15:56:20.849213 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:56:20.851803", "age": 3430.3142257680001, "duration": 3430.3148090620002, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348456034", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348456034 }, "events": [ { "time": "2021-04-21 15:56:20.851803", "event": "initiated" }, { "time": "2021-04-21 15:56:20.851803", "event": "header_read" }, { "time": "2021-04-21 15:56:20.851815", "event": "throttled" }, { "time": "2021-04-21 15:56:20.851853", "event": "all_read" }, { "time": "2021-04-21 15:56:20.851983", "event": "dispatched" }, { "time": "2021-04-21 16:01:40.084452", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084454", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9801215:348456035 lookup #0x10004cd0675/tasks 2021-04-21 15:56:20.849213 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:56:20.851899", "age": 3430.314129722, "duration": 3430.3147965909998, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348456035", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348456035 }, "events": [ { "time": "2021-04-21 15:56:20.851899", "event": "initiated" }, { "time": "2021-04-21 15:56:20.851899", "event": "header_read" }, { "time": "2021-04-21 15:56:20.851900", "event": "throttled" }, { "time": "2021-04-21 15:56:20.851908", "event": "all_read" }, { "time": "2021-04-21 15:56:20.852165", "event": "dispatched" }, { "time": "2021-04-21 15:56:20.852254", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:01:40.084455", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084457", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9801215:348456036 lookup #0x10004cd0675/slaves 2021-04-21 15:56:20.849213 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:56:20.879287", "age": 3430.2867415639998, "duration": 3430.287496596, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348456036", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348456036 }, "events": [ { "time": "2021-04-21 15:56:20.879287", "event": "initiated" }, { "time": "2021-04-21 15:56:20.879287", "event": "header_read" }, { "time": "2021-04-21 15:56:20.879292", "event": "throttled" }, { "time": "2021-04-21 15:56:20.879306", "event": "all_read" }, { "time": "2021-04-21 15:56:20.879408", "event": "dispatched" }, { "time": "2021-04-21 15:56:20.879502", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:01:40.084458", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084460", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9801215:348456042 lookup #0x10004cd0675/jul 2021-04-21 15:58:53.729411 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:58:53.740318", "age": 3277.4257106780001, "duration": 3277.4265688219998, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348456042", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348456042 }, "events": [ { "time": "2021-04-21 15:58:53.740318", "event": "initiated" }, { "time": "2021-04-21 15:58:53.740318", "event": "header_read" }, { "time": "2021-04-21 15:58:53.740324", "event": "throttled" }, { "time": "2021-04-21 15:58:53.740345", "event": "all_read" }, { "time": "2021-04-21 15:58:53.743424", "event": "dispatched" }, { "time": "2021-04-21 15:58:53.750527", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:01:40.084465", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084468", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9143865:2383914038 getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:29:35.332632 caller_uid=903, caller_gid=1003{})", "initiated_at": "2021-04-21 16:29:35.333838", "age": 1435.8321910940001, "duration": 1435.833136707, "type_data": { "flag_point": "failed to authpin, subtree is being exported", "reqid": "client.9143865:2383914038", "op_type": "client_request", "client_info": { "client": "client.9143865", "tid": 2383914038 }, "events": [ { "time": "2021-04-21 16:29:35.333838", "event": "initiated" }, { "time": "2021-04-21 16:29:35.333838", "event": "header_read" }, { "time": "2021-04-21 16:29:35.333841", "event": "throttled" }, { "time": "2021-04-21 16:29:35.333854", "event": "all_read" }, { "time": "2021-04-21 16:29:35.335013", "event": "dispatched" }, { "time": "2021-04-21 16:29:35.335091", "event": "failed to authpin, subtree is being exported" } ] } }, { "description": "client_request(client.10439445:7 lookup #0x10004cd0675/jul 2021-04-21 16:26:17.334272 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 16:26:17.352448", "age": 1633.813581226, "duration": 1633.8146036109999, "type_data": { "flag_point": "cleaned up request", "reqid": "client.10439445:7", "op_type": "client_request", "client_info": { "client": "client.10439445", "tid": 7 }, "events": [ { "time": "2021-04-21 16:26:17.352448", "event": "initiated" }, { "time": "2021-04-21 16:26:17.352448", "event": "header_read" }, { "time": "2021-04-21 16:26:17.352450", "event": "throttled" }, { "time": "2021-04-21 16:26:17.352460", "event": "all_read" }, { "time": "2021-04-21 16:26:17.356901", "event": "dispatched" }, { "time": "2021-04-21 16:26:17.357337", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:38:08.054109", "event": "killing request" }, { "time": "2021-04-21 16:38:08.054120", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9801215:348456041 lookup #0x10004cd0675/jul 2021-04-21 15:56:23.089216 caller_uid=1000, caller_gid=1000{})", "initiated_at": "2021-04-21 15:56:23.091531", "age": 3428.0744974170002, "duration": 3428.0756079930002, "type_data": { "flag_point": "cleaned up request", "reqid": "client.9801215:348456041", "op_type": "client_request", "client_info": { "client": "client.9801215", "tid": 348456041 }, "events": [ { "time": "2021-04-21 15:56:23.091531", "event": "initiated" }, { "time": "2021-04-21 15:56:23.091531", "event": "header_read" }, { "time": "2021-04-21 15:56:23.091536", "event": "throttled" }, { "time": "2021-04-21 15:56:23.091555", "event": "all_read" }, { "time": "2021-04-21 15:56:23.091660", "event": "dispatched" }, { "time": "2021-04-21 15:56:23.091763", "event": "failed to authpin, subtree is being exported" }, { "time": "2021-04-21 16:01:40.084461", "event": "killing request" }, { "time": "2021-04-21 16:01:40.084465", "event": "cleaned up request" } ] } }, { "description": "client_request(client.9143865:2383885743 getattr pAsLsXsFs #0x10004f6a16b 2021-04-21 16:26:36.713371 caller_uid=903, caller_gid=1003{})", "initiated_at": "2021-04-21 16:26:36.716418", "age": 1614.4496113350001, "duration": 1614.4508114350001, "type_data": { "flag_point": "failed to authpin, subtree is being exported", "reqid": "client.9143865:2383885743", "op_type": "client_request", "client_info": { "client": "client.9143865", "tid": 2383885743 }, "events": [ { "time": "2021-04-21 16:26:36.716418", "event": "initiated" }, { "time": "2021-04-21 16:26:36.716418", "event": "header_read" }, { "time": "2021-04-21 16:26:36.716421", "event": "throttled" }, { "time": "2021-04-21 16:26:36.716435", "event": "all_read" }, { "time": "2021-04-21 16:26:36.716526", "event": "dispatched" }, { "time": "2021-04-21 16:26:36.716632", "event": "failed to authpin, subtree is being exported" } ] } } ], "num_ops": 14 } -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

2 7

osd nearfull is not detected
by Konstantin Shalygin 21 Apr '21

21 Apr '21

Hi, On the adopted cluster Prometheus was triggered for "osd full > 90%" But Ceph itself - not. Actually OSD is drained (see %USE). root@host# ceph osd df name osd.696 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 696 nvme 0.91199 1.00000 912 GiB 830 GiB 684 GiB 8 KiB 146 GiB 81 GiB 91.09 1.00 47 up TOTAL 912 GiB 830 GiB 684 GiB 8.1 KiB 146 GiB 81 GiB 91.09 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 root@host# ceph osd df name osd.696 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 696 nvme 0.91199 1.00000 912 GiB 830 GiB 684 GiB 8 KiB 146 GiB 81 GiB 91.08 1.00 47 up TOTAL 912 GiB 830 GiB 684 GiB 8.1 KiB 146 GiB 81 GiB 91.08 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 root@host# ceph osd df name osd.696 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 696 nvme 0.91199 1.00000 912 GiB 830 GiB 684 GiB 8 KiB 146 GiB 81 GiB 91.07 1.00 47 up TOTAL 912 GiB 830 GiB 684 GiB 8.1 KiB 146 GiB 81 GiB 91.07 MIN/MAX VAR: 1.00/1.00 STDDEV: 0 Pool 18 is another class pool, OSD's of this pool triggered as usual, but for pool 17 - don't. root@host# ceph health detail HEALTH_WARN noout flag(s) set; Some pool(s) have the nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull OSDMAP_FLAGS noout flag(s) set POOL_SCRUB_FLAGS Some pool(s) have the nodeep-scrub flag(s) set Pool meta_ru1b has nodeep-scrub flag Pool data_ru1b has nodeep-scrub flag PG_BACKFILL_FULL Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 18.1008 is active+remapped+backfill_wait+backfill_toofull, acting [336,462,580] pg 18.27e0 is active+remapped+backfill_wait+backfill_toofull, acting [401,627,210] On my experience , Ceph triggers when OSD drain on backfillfull_ratio, then on nearfull_ratio until usage will drops to 84.99% I don't think is to possible to configure silence for this Current usage: root@host# ceph df detail RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.3 PiB 1022 TiB 3.3 PiB 3.3 PiB 76.71 nvme 161 TiB 61 TiB 82 TiB 100 TiB 62.30 TOTAL 4.4 PiB 1.1 PiB 3.4 PiB 3.4 PiB 76.20 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR meta_ru1b 17 2048 3.1 TiB 7.15G 82 TiB 92.77 2.1 TiB N/A N/A 7.15G 0 B 0 B data_ru1b 18 16384 1.1 PiB 3.07G 3.3 PiB 88.29 148 TiB N/A N/A 3.07G 0 B 0 B Current OSD dump header: epoch 270540 fsid ccf2c233-4adf-423c-b734-236220096d4e created 2019-02-14 15:30:56.642918 modified 2021-04-21 20:33:54.481616 flags noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 7255 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client jewel require_osd_release nautilus pool 17 'meta_ru1b' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode warn last_change 240836 lfor 0/0/51990 flags hashpspool,nodeep-scrub stripe_width 0 application metadata pool 18 'data_ru1b' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change 270529 lfor 0/0/52038 flags hashpspool,nodeep-scrub stripe_width 0 application data max_osd 780 Current versions: { "mon": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 780 }, "mds": {}, "overall": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 786 } } Dan, maybe there was something like that in your memory? My guess is that some counter type overflowed Thanks, k

2 2

New Ceph cluster- having issue with one monitor
by Robert W. Eckert 21 Apr '21

21 Apr '21

Hi, I have pieced together some pcs which I had been using to run a windows DFS cluster. the 3 servers all have 3 4Tb Hard Drives and 1 2Tb SSD, but they have different CPUs All of them are running RHEL8, and have 2.5 Gbps NICs in them. The install was with cephadm, and the ceph processes are all running under podman. Two of the servers - which are on old hardware, one a Core I5 2500, the other a J5005 embedded cpu are very stable, and have had no real issues. The other server an AMD 3600 on an X570 motherboard runs great, but the monitor goes down at least once a day. I have a script I run to copy from one of the other servers when it goes down, but I would prefer to just have it run. I don't know if it is tied together or not, but I also see OSD_SCRUB_ERRORS and PG_DAMAGED. The load on the cluster is light-I have some backups from my pcs running to it, a copy of all my photos, 3 private Minecraft servers running in podman containers, and an ark survival server also running in a podman container with a mount pointing to the ceph file system. Last night, I stopped the Minecraft and ark containers before I went to bed, and this morning, everything was clean. This afternoon, with no load on the filesystem, the monitor went down, and I have 8 scrub errors. I have not been able to find where the ceph monitor logs are going, I thought under either /var/log/ceph or /var/log/ceph/fe3a7cb0-69ca-11eb-8d45-c86000d08867 But the log files there are 0 bytes. Attached is the output from ceph-report. What can I try to improve logging for troubleshooting or get this one monitor stable? Thanks, Rob

1 0

MDS replay takes forever and cephfs is down
by Flemming Frandsen 21 Apr '21

21 Apr '21

I tried restarting an MDS server using: systemctl restart ceph-mds(a)ardmore.service This caused the standby server to enter replay state and the fs started hanging for several minutes. In a slight panic I restarted the other mds server, which was replaced by the standby server and it almost immediately entered resolve state. fs dump shows a seq number counting upwards very slowly for the replay'ing MDS server, I have no idea how far it needs to count: # ceph fs dump dumped fsmap epoch 1030314 e1030314 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in sep arate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 1030314 flags 12 created 2019-09-09 13:08:26.830927 modified 2021-04-21 14:04:14.672440 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 min_compat_client -1 (unspecified) last_failure 0 last_failure_osd_epoch 13610 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in sep arate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 2 in 0,1 up {0=10398946,1=10404857} failed damaged stopped data_pools [1] metadata_pool 2 inline_data disabled balancer standby_count_wanted 1 [mds.dalmore{0:10398946} state up:replay seq 215 addr [v2: 10.0.37.222:6800/2681188441,v1:10.0.37.222:6801/2681188441]] [mds.cragganmore{1:10404857} state up:resolve seq 201 addr [v2: 10.0.37.221:6800/871249119,v1:10.0.37.221:6801/871249119]] Standby daemons: [mds.ardmore{-1:10408652} state up:standby seq 2 addr [v2: 10.0.37.223:6800/4096598841,v1:10.0.37.223:6801/4096598841]] Earlier today we added a new OSD host with 12 new OSDs and backfilling is proceeding as expected: cluster: id: e2007417-a346-4af7-8aa9-4ce8f0d73661 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs behind on trimming services: mon: 3 daemons, quorum cragganmore,dalmore,ardmore (age 5w) mgr: ardmore(active, since 2w), standbys: dalmore, cragganmore mds: cephfs:2/2 {0=dalmore=up:replay,1=cragganmore=up:resolve} 1 up:standby osd: 69 osds: 69 up (since 102m), 69 in (since 102m); 443 remapped pgs rgw: 9 daemons active (ardmore.rgw0, ardmore.rgw1, ardmore.rgw2, cragganmore.rgw0, cragganmore.rgw1, cragganmore.rgw2, dalmore.rgw0, dalmore.rgw1, dalmore.rgw2) task status: scrub status: mds.cragganmore: idle mds.dalmore: idle data: pools: 13 pools, 1440 pgs objects: 50.57M objects, 9.0 TiB usage: 34 TiB used, 37 TiB / 71 TiB avail pgs: 30195420/151707033 objects misplaced (19.904%) 997 active+clean 431 active+remapped+backfill_wait 12 active+remapped+backfilling io: client: 65 MiB/s rd, 206 KiB/s wr, 17 op/s rd, 8 op/s wr recovery: 5.5 MiB/s, 23 objects/s progress: Rebalancing after osd.62 marked in [======================........] Rebalancing after osd.67 marked in [===========...................] Rebalancing after osd.68 marked in [============..................] Rebalancing after osd.64 marked in [=====================.........] Rebalancing after osd.60 marked in [====================..........] Rebalancing after osd.66 marked in [=============.................] Rebalancing after osd.63 marked in [=====================.........] Rebalancing after osd.61 marked in [======================........] Rebalancing after osd.59 marked in [======================........] Rebalancing after osd.58 marked in [========================......] Rebalancing after osd.57 marked in [===========================...] Rebalancing after osd.65 marked in [==================............] It seems we're running a mix of versions: ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 57, "ceph version 14.2.20 (36274af6eb7f2a5055f2d53ad448f2694e9046a0) nautilus (stable)": 12 }, "mds": { "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 3 }, "rgw": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 9 }, "overall": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 69, "ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable)": 6, "ceph version 14.2.20 (36274af6eb7f2a5055f2d53ad448f2694e9046a0) nautilus (stable)": 12 } } Any hints will be greatly appreciated. -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

2 2

Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
by Szabo, Istvan (Agoda) 21 Apr '21

21 Apr '21

Hi Ceph Users, Here is the latest request I tried but still not working curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<password>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx000000000000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-data</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Swift Stat Timeout
by Dylan Griff 21 Apr '21

21 Apr '21

Hey folks! We have a user with ~1900 buckets in our RGW service and running this stat command results in a timeout for them: swift -A https://<URL>:443/auth/1.0 -U <UID> -K <KEY> stat Running the same command, but specifiying one of their buckets, returns promptly. Running the command for a different user with minimal buckets returns promptly as well. Turning up debug logging to 20 for rgw resulted in a great deal of logs showing: 20 reading from default.rgw.meta:root:.bucket.meta.<BUCKET-ID> 20 get_system_obj_state: rctx=0x559b32a6b570 obj=default.rgw.meta:root:.bucket.meta.<BUCKET-ID> state=0x559b32c37e40 s->prefetch_data=0 10 cache get: name=default.rgw.meta+root+.bucket.meta.<BUCKET-ID> : hit (requested=0x16, cached=0x17) 20 get_system_obj_state: s->obj_tag was set empty 10 cache get: name=default.rgw.meta+root+.bucket.meta.<BUCKET-ID> : hit (requested=0x11, cached=0x17) Which looks like to me it is iterating getting the state of all their stuff. My question: is ~1900 an unreasonable amount of buckets such that we should expect to see this full account 'stat' command timeout? Or should I be expecting it to return promptly still? Thanks! Cheers, Dylan -- Dylan Griff Senior System Administrator CLE D063 RCS - Systems - University of Victoria

1 2

HBA vs caching Raid controller
by Nico Schottelius 21 Apr '21

21 Apr '21

Good evening, I've to tackle an old, probably recurring topic: HBAs vs. Raid controllers. While generally speaking many people in the ceph field recommend to go with HBAs, it seems in our infrastructure the only server we phased in with an HBA vs. raid controller is actually doing worse in terms of latency. For the background: we have many Perc H800+MD1200 [1] systems running with 10TB HDDs (raid0, read ahead, writeback cache). One server has LSI SAS3008 [0] instead of the Perc H800, which comes with 512MB RAM + BBU. On most servers latencies are around 4-12ms (average 6ms), on the system with the LSI controller we see 20-60ms (average 30ms) latency. Now, my question is, are we doing some inherently wrong with the SAS3008 or does in fact the cache help to possible reduce seek time? We were considering to move more towards LSI HBAs to reduce maintenance effort, however if we have a factor of 5 in latency between the two different systems, it might be better to stay on the H800 path for disks. Any input/experiences appreciated. Best regards, Nico [0] 05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: Dell 12Gbps HBA Kernel driver in use: mpt3sas Kernel modules: mpt3sas [1] 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) Subsystem: Dell PERC H800 Adapter Kernel driver in use: megaraid_sas Kernel modules: megaraid_sas -- Sustainable and modern Infrastructures by ungleich.ch

5 12

ceph orch upgrade fails when pulling container image
by Robert Sander 21 Apr '21

21 Apr '21

Hi, # docker pull ceph/ceph:v16.2.1 Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit How do I update a Ceph cluster in this situation? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2