May 2021 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

OSD slow ops warning not clearing after OSD down
by Frank Schilder 17 Jan '23

17 Jan '23

Dear cephers, I have a strange problem. An OSD went down and recovery finished. For some reason, I have a slow ops warning for the failed OSD stuck in the system: health: HEALTH_WARN 430 slow ops, oldest one blocked for 36 sec, osd.580 has slow ops The OSD is auto-out: | 580 | ceph-22 | 0 | 0 | 0 | 0 | 0 | 0 | autoout,exists | It is probably a warning dating back to just before the fail. How can I clear it? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 7

[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 9

Upgrade tips from Luminous to Nautilus?
by Mark Schouten 08 Jul '21

08 Jul '21

Hi, We've done our fair share of Ceph cluster upgrades since Hammer, and have not seen much problems with them. I'm now at the point that I have to upgrade a rather large cluster running Luminous and I would like to hear from other users if they have experiences with issues I can expect so that I can anticipate on them beforehand. As said, the cluster is running Luminous (12.2.13) and has the following services active: services: mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 mgr: osdnode01(active), standbys: osdnode02, osdnode03 mds: pmrb-3/3/3 up {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 up:standby osd: 116 osds: 116 up, 116 in; rgw: 3 daemons active Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster is 1.01PiB. We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is a total of 5760 pgs. { "rule_id": 1, "rule_name": "hdd-data", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "ssd-data", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } rbd -> crush_rule: hdd-data .rgw.root -> crush_rule: hdd-data default.rgw.control -> crush_rule: hdd-data default.rgw.data.root -> crush_rule: ssd-data default.rgw.gc -> crush_rule: ssd-data default.rgw.log -> crush_rule: ssd-data default.rgw.users.uid -> crush_rule: hdd-data default.rgw.usage -> crush_rule: ssd-data default.rgw.users.email -> crush_rule: hdd-data default.rgw.users.keys -> crush_rule: hdd-data default.rgw.meta -> crush_rule: hdd-data default.rgw.buckets.index -> crush_rule: ssd-data default.rgw.buckets.data -> crush_rule: hdd-data default.rgw.users.swift -> crush_rule: hdd-data default.rgw.buckets.non-ec -> crush_rule: ssd-data DB0475 -> crush_rule: hdd-data cephfs_pmrb_data -> crush_rule: hdd-data cephfs_pmrb_metadata -> crush_rule: ssd-data All but four clients are running Luminous, the four are running Jewel (that needs upgrading before proceeding with this upgrade). So, normally, I would 'just' upgrade all Ceph packages on the monitor-nodes and restart mons and then mgrs. After that, I would upgrade all Ceph packages on the OSD nodes and restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting the OSD's will probably take a while. If anyone has a hint on what I should expect to cause some extra load or waiting time, that would be great. Obviously, we have read https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking for real world experiences. Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

4 11

Re: XFS on RBD on EC painfully slow
by Sebastian Knust 06 Jul '21

06 Jul '21

Hi Reed, To add to this command by Weiwen: On 28.05.21 13:03, 胡玮文 wrote: > Have you tried just start multiple rsync process simultaneously to transfer different directories? Distributed system like ceph often benefits from more parallelism. When I migrated from XFS on iSCSI (legacy system, no Ceph) to CephFS a few months ago, I used msrsync [1] and was quite happy with the speed. For your use case, I would start with -p 12 but might experiment with up to -p 24 (as you only have 6C/12T in your CPU). With many small files, you also might want to increase -s from the default 1000. Note that msrsync does not work with the --delete rsync flag. As I was syncing a live system, I ended up with this workflow: - Initial sync with msrsync (something like ./msrsync -p 12 --progress --stats --rsync "-aS --numeric-ids" ...) - Second sync with msrsync (to sync changes during the first sync) - Take old storage off-line for users / read-only - Final rsync with --delete (i.e. rsync -aS --numeric-ids --delete ...) - Mount cephfs at location of old storage, adjust /etc/exports with fsid entries where necessary, turn system back on-line / read-write Cheers Sebastian [1] https://github.com/jbd/msrsync

3 2

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

ceph-Dokan on windows 10 not working after upgrade to pacific
by Robert W. Eckert 29 Jun '21

29 Jun '21

Hi- I recently upgraded to pacific, and I am now getting an error connecting on my windows 10 machine: The error is the handle_auth_bad_method, I tried a few combinations of cephx,none on the monitors, but I keep getting the same error. The same config(With paths updated) and key ring works on my WSL instance running an old luminous client (I can't seem to get it to install a newer client ) Do you have any suggestions on where to look? Thanks, Rob. ----------------------------------------------------- PS C:\Program Files\Ceph\bin> .\ceph-dokan.exe --id rob -l Q 2021-05-14T12:19:58.172Eastern Daylight Time 5 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] failed to fetch mon config (--no-mon-config to skip) PS C:\Program Files\Ceph\bin> cat c:/ProgramData/ceph/ceph.client.rob.keyring [client.rob] key = <REDACTED> caps mon = "allow rwx" caps osd = "allow rwx" PS C:\Program Files\Ceph\bin> cat C:\ProgramData\Ceph\ceph.conf # minimal ceph.conf [global] log to stderr = true ; Uncomment the following in order to use the Windows Event Log log to syslog = true run dir = C:/ProgramData/ceph/out crash dir = C:/ProgramData/ceph/out ; Use the following to change the cephfs client log level debug client = 2 [global] fsid = <redacted> mon_host = [<redacted>] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx [client] keyring = c:/ProgramData/ceph/ceph.client.rob.keyring log file = C:/ProgramData/ceph/out/$name.$pid.log admin socket = C:/ProgramData/ceph/out/$name.$pid.asok

2 1

Ceph Month June Schedule Now Available
by Mike Perez 24 Jun '21

24 Jun '21

Hi everyone, The Ceph Month June schedule is now available: https://pad.ceph.com/p/ceph-month-june-2021 We have great sessions from component updates, performance best practices, Ceph on different architectures, BoF sessions to get more involved with working groups in the community, and more! You may also leave open discussion topics for the listed talks that we'll get to each Q/A portion. I will provide the video stream link on this thread and etherpad once it's available. You can also add the Ceph community calendar, which will have the Ceph Month sessions prefixed with "Ceph Month" to get local timezone conversions. https://calendar.google.com/calendar/embed?src=9ts9c7lt7u1vic2ijvvqqlfpo0%4… Thank you to our speakers for taking the time to share with us all the latest best practices and usage with Ceph! -- Mike Perez

2 13

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 16 Jun '21

16 Jun '21

On Mon, May 3, 2021 at 12:27 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 12:25 schrieb Ilya Dryomov: > > ceph osd setmaxosd 10 > > Bingo! Mount works again. > > Veeeery strange things are going on here (-: > > Thanx a lot for now!! If I can help to track it down, please let me know. Good to know it helped! I'll think about this some more and probably plan to patch the kernel client to be less stringent and not choke on this sort of misconfiguration. Thanks, Ilya

3 6

Re: ceph df: pool stored vs bytes_used -- raw or not?
by Konstantin Shalygin 15 Jun '21

15 Jun '21

> On 20 May 2021, at 19:47, Igor Fedotov <ifedotov(a)suse.de> wrote: > > which PR/ticket are you referring to? This thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW…> And this ticket https://tracker.ceph.com/issues/48385 <https://tracker.ceph.com/issues/48385> All pools show me stored == used, even if all osd in/up/weighted on 14.2.21 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 595 TiB 334 TiB 261 TiB 261 TiB 43.87 nvme 52 TiB 34 TiB 18 TiB 18 TiB 34.68 TOTAL 647 TiB 368 TiB 279 TiB 279 TiB 43.13 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL replicated_rbd 1 1024 29 TiB 7.74M 29 TiB 10.93 80 TiB erasure_rbd_meta 6 16 2.9 MiB 154 2.9 MiB 0 7.3 TiB erasure_rbd_data 7 512 44 TiB 11.64M 44 TiB 15.61 144 TiB .rgw.root 12 16 11 KiB 22 11 KiB 0 80 TiB default.rgw.control 13 16 0 B 8 0 B 0 80 TiB default.rgw.meta 14 16 45 KiB 211 45 KiB 0 7.3 TiB default.rgw.log 15 16 88 MiB 345 88 MiB 0 80 TiB default.rgw.buckets.index 16 16 6.3 GiB 717 6.3 GiB 0.03 7.3 TiB default.rgw.buckets.non-ec 17 16 810 KiB 5.74k 810 KiB 0 80 TiB default.rgw.buckets.data 18 512 15 TiB 33.99M 15 TiB 5.70 80 TiB replicated_rbd_nvme 19 256 3.3 TiB 880.95k 3.3 TiB 13.25 7.3 TiB unredundant_rbd_nvme 20 128 3.9 TiB 1.03M 3.9 TiB 15.27 11 TiB fs_data 21 256 11 TiB 12.32M 11 TiB 4.38 80 TiB fs_meta 22 16 4.8 GiB 636.79k 4.8 GiB 0.02 7.3 TiB fs_data_nvme 23 64 20 GiB 5.97k 20 GiB 0.09 7.3 TiB kubernetes_rbd 24 32 142 GiB 36.71k 142 GiB 0.06 80 TiB Thanks, k

3 4

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Bucket creation on RGW Multisite env.
by mhnx 08 Jun '21

08 Jun '21

Hello. I have a multisite RGW environment. When I create a new bucket, the bucket is directly created on master and secondary. If I don't want to sync a bucket, I need to stop sync after creation. Is there any global option as "Do not sync directly, only start if I want to" ?

3 6

Re: SSD recommendations for RBD and VM's
by mhnx 06 Jun '21

06 Jun '21

Hello Samuel. Thanks for the answer. Yes the Intel S4510 series is a good choice but it's expensive. I have 21 server and data distribution is quite well. At power loss I don't think I'll lose data. All the VM's using same image and the rest is cookie. In this case I'm not sure I should spend extra money on PLP. Actually I like Samsung 870 EVO. It's cheap and I think 300TBW will be enough for 5-10years. Do you know any better ssd with the same price range as 870 EVO? Samsung 870 evo (500GB) = 5 Years or 300 TBW - $64.99 Samsung 860 pro (512GB) = 5 Years or 600 TBW - $99 > > I would recommend Intel S4510 series, which has power loss protection (PLP). > > If you do not care about PLP, lower-cost Samsung 870EVO and Crucial MX500 should also be OK (with separate DB/WAL on enterprise SSD with PLP) > > Samuel > > ________________________________ > huxiaoyu(a)horebdata.cn > > > From: by morphin > Date: 2021-05-30 02:48 > To: Anthony D'Atri > CC: Ceph Users > Subject: [ceph-users] Re: SSD recommendations for RBD and VM's > Hello Anthony. > > I use Qemu and I don't need size. > I've 1000 vm and usually they're clones from the same rbd image. The > image is 30GB. > Right now I've 7TB Stored data. rep x3 = 20TB data. It's mostly read > intensive. Usage is stable and does not grow. > So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's. > I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB > will be enough. Actually 120PCS 256GB will be better but the price > goes up. > I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've > 40PCS 100GB. I'm gonna make them OSD as well. > 7 years and DC S3700 still rocks. Not even one of them is dead. > The SSD must be Low price & High TBW life span. Rest is not important. > > > Anthony D'Atri <anthony.datri(a)gmail.com>, 30 May 2021 Paz, 02:26 > tarihinde şunu yazdı: > > > > The choice depends on scale, your choice of chassis / form factor, budget, workload and needs. > > > > The sizes you list seem awfully small. Tell us more about your use-case. OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ? > > —aad > > > > > > > On May 29, 2021, at 2:10 PM, by morphin <morphinwithyou(a)gmail.com> wrote: > > > > > > Hello. > > > > > > I have virtualization env and I'm looking new SSD for HDD replacement. > > > What are the best Performance / Price SSDs in the market right now? > > > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. > > > > > > Is there a SSD recommendation list for ceph? > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

5 9

CRUSH rule for EC 6+2 on 6-node cluster
by Bryan Stillwell 04 Jun '21

04 Jun '21

I'm trying to figure out a CRUSH rule that will spread data out across my cluster as much as possible, but not more than 2 chunks per host. If I use the default rule with an osd failure domain like this: step take default step choose indep 0 type osd step emit I get clustering of 3-4 chunks on some of the hosts: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do > echo $pg > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do > ceph osd find $osd | jq -r '.host' > done | sort | uniq -c | sort -n -k1 8.0 1 harrahs 3 paris 4 aladdin 8.1 1 aladdin 1 excalibur 2 mandalaybay 4 paris 8.2 1 harrahs 2 aladdin 2 mirage 3 paris ... However, if I change the rule to use: step take default step choose indep 0 type host step chooseleaf indep 2 type osd step emit I get the data spread across 4 hosts with 2 chunks per host: # for pg in $(ceph pg ls-by-pool cephfs_data_ec62 -f json | jq -r '.pg_stats[].pgid'); do > echo $pg > for osd in $(ceph pg map $pg -f json | jq -r '.up[]'); do > ceph osd find $osd | jq -r '.host' > done | sort | uniq -c | sort -n -k1 > done 8.0 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.1 2 aladdin 2 harrahs 2 mandalaybay 2 paris 8.2 2 harrahs 2 mandalaybay 2 mirage 2 paris ... Is it possible to get the data to spread out over more hosts? I plan on expanding the cluster in the near future and would like to see more hosts get 1 chunk instead of 2. Also, before you recommend adding two more hosts and switching to a host-based failure domain, the cluster is on a variety of hardware with between 2-6 drives per host and drives that are 4TB-12TB in size (it's part of my home lab). Thanks, Bryan

5 12

MDS cache tunning
by Andres Rojas Guerrero 02 Jun '21

02 Jun '21

Hi all, I have observed that the MDS Cache Configuration has 18 parameters: mds_cache_memory_limit mds_cache_reservation mds_health_cache_threshold mds_cache_trim_threshold mds_cache_trim_decay_rate mds_recall_max_caps mds_recall_max_decay_threshold mds_recall_max_decay_rate mds_recall_global_max_decay_threshold mds_recall_warning_threshold mds_recall_warning_decay_rate mds_session_cap_acquisition_throttle mds_session_cap_acquisition_decay_rate mds_session_max_caps_throttle_ratio mds_cap_acquisition_throttle_retry_request_timeout mds_session_cache_liveness_magnitude mds_session_cache_liveness_decay_rate mds_max_caps_per_client I find the Ceph documentation in this section a bit cryptic and I have tried to find some resources that talk about how to tune these parameters, but without success. Does anyone have experience in adjusting these parameters according to the characteristics of the Ceph cluster itself, the hardware and the use of MDS? Regards! -- ******************************************************* Andrés Rojas Guerrero Unidad Sistemas Linux Area Arquitectura Tecnológica Secretaría General Adjunta de Informática Consejo Superior de Investigaciones Científicas (CSIC) Pinar 19 28006 - Madrid Tel: +34 915680059 -- Ext. 990059 email: a.rojas(a)csic.es ID comunicate.csic.es: @50852720l:matrix.csic.es *******************************************************

2 11

Fwd: Re: Ceph osd will not start.
by Marco Pizzolo 01 Jun '21

01 Jun '21

Peter, We're seeing the same issues as you are. We have 2 new hosts Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz w/ 48 cores, 384GB RAM, and 60x 10TB SED drives and we have tried both 15.2.13 and 16.2.4 Cephadm does NOT properly deploy and activate OSDs on Ubuntu 20.04.2 with Docker. Seems to be a bug in Cephadm and a product regression, as we have 4 near identical nodes on Centos running Nautilus (240 x 10TB SED drives) and had no problems. FWIW we had no luck yet with one-by-one OSD daemon additions through ceph orch either. We also reproduced the issue easily in a virtual lab using small virtual disks on a single ceph VM with 1 mon. We are now looking into whether we can get past this with a manual buildout. If you, or anyone, has hit the same stumbling block and gotten past it, I would really appreciate some guidance. Thanks, Marco On Thu, May 27, 2021 at 2:23 PM Peter Childs <pchilds(a)bcs.org> wrote: > In the end it looks like I might be able to get the node up to about 30 > odds before it stops creating any more. > > Or more it formats the disks but freezes up starting the daemons. > > I suspect I'm missing somthing I can tune to get it working better. > > If I could see any error messages that might help, but I'm yet to spit > anything. > > Peter. > > On Wed, 26 May 2021, 10:57 Eugen Block, <eblock(a)nde.ag> wrote: > > > > If I add the osd daemons one at a time with > > > > > > ceph orch daemon add osd drywood12:/dev/sda > > > > > > It does actually work, > > > > Great! > > > > > I suspect what's happening is when my rule for creating osds run and > > > creates them all-at-once it ties the orch it overloads cephadm and it > > can't > > > cope. > > > > It's possible, I guess. > > > > > I suspect what I might need to do at least to work around the issue is > > set > > > "limit:" and bring it up until it stops working. > > > > It's worth a try, yes, although the docs state you should try to avoid > > it, it's possible that it doesn't work properly, in that case create a > > bug report. ;-) > > > > > I did work out how to get ceph-volume to nearly work manually. > > > > > > cephadm shell > > > ceph auth get client.bootstrap-osd -o > > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > > ceph-volume lvm create --data /dev/sda --dmcrypt > > > > > > but given I've now got "add osd" to work, I suspect I just need to fine > > > tune my osd creation rules, so it does not try and create too many osds > > on > > > the same node at the same time. > > > > I agree, no need to do it manually if there is an automated way, > > especially if you're trying to bring up dozens of OSDs. > > > > > > Zitat von Peter Childs <pchilds(a)bcs.org>: > > > > > After a bit of messing around. I managed to get it somewhat working. > > > > > > If I add the osd daemons one at a time with > > > > > > ceph orch daemon add osd drywood12:/dev/sda > > > > > > It does actually work, > > > > > > I suspect what's happening is when my rule for creating osds run and > > > creates them all-at-once it ties the orch it overloads cephadm and it > > can't > > > cope. > > > > > > service_type: osd > > > service_name: osd.drywood-disks > > > placement: > > > host_pattern: 'drywood*' > > > spec: > > > data_devices: > > > size: "7TB:" > > > objectstore: bluestore > > > > > > I suspect what I might need to do at least to work around the issue is > > set > > > "limit:" and bring it up until it stops working. > > > > > > I did work out how to get ceph-volume to nearly work manually. > > > > > > cephadm shell > > > ceph auth get client.bootstrap-osd -o > > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > > ceph-volume lvm create --data /dev/sda --dmcrypt > > > > > > but given I've now got "add osd" to work, I suspect I just need to fine > > > tune my osd creation rules, so it does not try and create too many osds > > on > > > the same node at the same time. > > > > > > > > > > > > On Wed, 26 May 2021 at 08:25, Eugen Block <eblock(a)nde.ag> wrote: > > > > > >> Hi, > > >> > > >> I believe your current issue is due to a missing keyring for > > >> client.bootstrap-osd on the OSD node. But even after fixing that > > >> you'll probably still won't be able to deploy an OSD manually with > > >> ceph-volume because 'ceph-volume activate' is not supported with > > >> cephadm [1]. I just tried that in a virtual environment, it fails when > > >> activating the systemd-unit: > > >> > > >> ---snip--- > > >> [2021-05-26 06:47:16,677][ceph_volume.process][INFO ] Running > > >> command: /usr/bin/systemctl enable > > >> ceph-volume@lvm-8-1a8fc8ae-8f4c-4f91-b044-d5636bb52456 > > >> [2021-05-26 06:47:16,692][ceph_volume.process][INFO ] stderr Failed > > >> to connect to bus: No such file or directory > > >> [2021-05-26 06:47:16,693][ceph_volume.devices.lvm.create][ERROR ] lvm > > >> activate was unable to complete, while creating the OSD > > >> Traceback (most recent call last): > > >> File > > >> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", > > >> line 32, in create > > >> Activate([]).activate(args) > > >> File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", > > >> line 16, in is_root > > >> return func(*a, **kw) > > >> File > > >> > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py", > > >> line > > >> 294, in activate > > >> activate_bluestore(lvs, args.no_systemd) > > >> File > > >> > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py", > > >> line > > >> 214, in activate_bluestore > > >> systemctl.enable_volume(osd_id, osd_fsid, 'lvm') > > >> File > > >> "/usr/lib/python3.6/site-packages/ceph_volume/systemd/systemctl.py", > > >> line 82, in enable_volume > > >> return enable(volume_unit % (device_type, id_, fsid)) > > >> File > > >> "/usr/lib/python3.6/site-packages/ceph_volume/systemd/systemctl.py", > > >> line 22, in enable > > >> process.run(['systemctl', 'enable', unit]) > > >> File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", > > >> line 153, in run > > >> raise RuntimeError(msg) > > >> RuntimeError: command returned non-zero exit status: 1 > > >> [2021-05-26 06:47:16,694][ceph_volume.devices.lvm.create][INFO ] will > > >> rollback OSD ID creation > > >> [2021-05-26 06:47:16,697][ceph_volume.process][INFO ] Running > > >> command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > > >> --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.8 > > >> --yes-i-really-mean-it > > >> [2021-05-26 06:47:17,597][ceph_volume.process][INFO ] stderr purged > > osd.8 > > >> ---snip--- > > >> > > >> There's a workaround described in [2] that's not really an option for > > >> dozens of OSDs. I think your best approach is to bring cephadm to > > >> activate the OSDs for you. > > >> You wrote you didn't find any helpful error messages, but did cephadm > > >> even try to deploy OSDs? What does your osd spec file look like? Did > > >> you explicitly run 'ceph orch apply osd -i specfile.yml'? This should > > >> trigger cephadm and you should see at least some output like this: > > >> > > >> Mai 26 08:21:48 pacific1 conmon[31446]: 2021-05-26T06:21:48.466+0000 > > >> 7effc15ff700 0 log_channel(cephadm) log [INF] : Applying service > > >> osd.ssd-hdd-mix on host pacific2... > > >> Mai 26 08:21:49 pacific1 conmon[31009]: cephadm > > >> 2021-05-26T06:21:48.469611+0000 mgr.pacific1.whndiw (mgr.14166) 1646 : > > >> cephadm [INF] Applying service osd.ssd-hdd-mix on host pacific2... > > >> > > >> Regards, > > >> Eugen > > >> > > >> [1] https://tracker.ceph.com/issues/49159 > > >> [2] https://tracker.ceph.com/issues/46691 > > >> > > >> > > >> Zitat von Peter Childs <pchilds(a)bcs.org>: > > >> > > >> > Not sure what I'm doing wrong, I suspect its the way I'm running > > >> > ceph-volume. > > >> > > > >> > root@drywood12:~# cephadm ceph-volume lvm create --data /dev/sda > > >> --dmcrypt > > >> > Inferring fsid 1518c8e0-bbe4-11eb-9772-001e67dc85ea > > >> > Using recent ceph image ceph/ceph@sha256 > > >> > :54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > > >> > /usr/bin/docker: Running command: /usr/bin/ceph-authtool > > --gen-print-key > > >> > /usr/bin/docker: Running command: /usr/bin/ceph-authtool > > --gen-print-key > > >> > /usr/bin/docker: --> RuntimeError: No valid ceph configuration file > > was > > >> > loaded. > > >> > Traceback (most recent call last): > > >> > File "/usr/sbin/cephadm", line 8029, in <module> > > >> > main() > > >> > File "/usr/sbin/cephadm", line 8017, in main > > >> > r = ctx.func(ctx) > > >> > File "/usr/sbin/cephadm", line 1678, in _infer_fsid > > >> > return func(ctx) > > >> > File "/usr/sbin/cephadm", line 1738, in _infer_image > > >> > return func(ctx) > > >> > File "/usr/sbin/cephadm", line 4514, in command_ceph_volume > > >> > out, err, code = call_throws(ctx, c.run_cmd(), > > verbosity=verbosity) > > >> > File "/usr/sbin/cephadm", line 1464, in call_throws > > >> > raise RuntimeError('Failed command: %s' % ' '.join(command)) > > >> > RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host > > >> > --net=host --entrypoint /usr/sbin/ceph-volume --privileged > > >> --group-add=disk > > >> > --init -e CONTAINER_IMAGE=ceph/ceph@sha256 > :54e95ae1e11404157d7b329d0t > > >> > > > >> > root@drywood12:~# cephadm shell > > >> > Inferring fsid 1518c8e0-bbe4-11eb-9772-001e67dc85ea > > >> > Inferring config > > >> > > > /var/lib/ceph/1518c8e0-bbe4-11eb-9772-001e67dc85ea/mon.drywood12/config > > >> > Using recent ceph image ceph/ceph@sha256 > > >> > :54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > > >> > root@drywood12:/# ceph-volume lvm create --data /dev/sda --dmcrypt > > >> > Running command: /usr/bin/ceph-authtool --gen-print-key > > >> > Running command: /usr/bin/ceph-authtool --gen-print-key > > >> > Running command: /usr/bin/ceph --cluster ceph --name > > client.bootstrap-osd > > >> > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > > >> > 70054a5c-c176-463a-a0ac-b44c5db0987c > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable > to > > >> find > > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such > > file > > >> or > > >> > directory > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 > > >> > AuthRegistry(0x7fdef405b378) no keyring found at > > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable > to > > >> find > > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such > > file > > >> or > > >> > directory > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 > > >> > AuthRegistry(0x7fdef405ef20) no keyring found at > > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 auth: unable > to > > >> find > > >> > a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such > > file > > >> or > > >> > directory > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 > > >> > AuthRegistry(0x7fdef8f0bea0) no keyring found at > > >> > /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef2d9d700 -1 > > monclient(hunting): > > >> > handle_auth_bad_method server allowed_methods [2] but i only support > > [1] > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef259c700 -1 > > monclient(hunting): > > >> > handle_auth_bad_method server allowed_methods [2] but i only support > > [1] > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef1d9b700 -1 > > monclient(hunting): > > >> > handle_auth_bad_method server allowed_methods [2] but i only support > > [1] > > >> > stderr: 2021-05-25T07:46:18.188+0000 7fdef8f0d700 -1 monclient: > > >> > authenticate NOTE: no keyring found; disabled cephx authentication > > >> > stderr: [errno 13] RADOS permission denied (error connecting to the > > >> > cluster) > > >> > --> RuntimeError: Unable to create a new OSD id > > >> > root@drywood12:/# lsblk /dev/sda > > >> > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > > >> > sda 8:0 0 7.3T 0 disk > > >> > > > >> > As far as I can see cephadm gets a little further than this as the > > disks > > >> > have lvm volumes on them just the osd's daemons are not created or > > >> started. > > >> > So maybe I'm invoking ceph-volume incorrectly. > > >> > > > >> > > > >> > On Tue, 25 May 2021 at 06:57, Peter Childs <pchilds(a)bcs.org> wrote: > > >> > > > >> >> > > >> >> > > >> >> On Mon, 24 May 2021, 21:08 Marc, <Marc(a)f1-outsourcing.eu> wrote: > > >> >> > > >> >>> > > > >> >>> > I'm attempting to use cephadm and Pacific, currently on debian > > >> buster, > > >> >>> > mostly because centos7 ain't supported any more and cenotos8 > ain't > > >> >>> > support > > >> >>> > by some of my hardware. > > >> >>> > > >> >>> Who says centos7 is not supported any more? Afaik centos7/el7 is > > being > > >> >>> supported till its EOL 2024. By then maybe a good alternative for > > >> >>> el8/stream has surfaced. > > >> >>> > > >> >> > > >> >> Not supported by ceph Pacific, it's our os of choice otherwise. > > >> >> > > >> >> My testing says the version available of podman, docker and > python3, > > do > > >> >> not work with Pacific. > > >> >> > > >> >> Given I've needed to upgrade docker on buster can we please have a > > list > > >> of > > >> >> versions that work with cephadm, maybe even have cephadm say no, > > please > > >> >> upgrade unless your running the right version or better. > > >> >> > > >> >> > > >> >> > > >> >>> > Anyway I have a few nodes with 59x 7.2TB disks but for some > reason > > >> the > > >> >>> > osd > > >> >>> > daemons don't start, the disks get formatted and the osd are > > created > > >> but > > >> >>> > the daemons never come up. > > >> >>> > > >> >>> what if you try with > > >> >>> ceph-volume lvm create --data /dev/sdi --dmcrypt ? > > >> >>> > > >> >> > > >> >> I'll have a go. > > >> >> > > >> >> > > >> >>> > They are probably the wrong spec for ceph (48gb of memory and > > only 4 > > >> >>> > cores) > > >> >>> > > >> >>> You can always start with just configuring a few disks per node. > > That > > >> >>> should always work. > > >> >>> > > >> >> > > >> >> That was my thought too. > > >> >> > > >> >> Thanks > > >> >> > > >> >> Peter > > >> >> > > >> >> > > >> >>> > but I was expecting them to start and be either dirt slow or > crash > > >> >>> > later, > > >> >>> > anyway I've got upto 30 of them, so I was hoping on getting at > > least > > >> get > > >> >>> > 6PB of raw storage out of them. > > >> >>> > > > >> >>> > As yet I've not spotted any helpful error messages. > > >> >>> > > > >> >>> _______________________________________________ > > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io > > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >>> > > >> >> > > >> > _______________________________________________ > > >> > ceph-users mailing list -- ceph-users(a)ceph.io > > >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> > > >> > > >> _______________________________________________ > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 11

The always welcomed large omap
by Szabo, Istvan (Agoda) 01 Jun '21

01 Jun '21

Hi, Any way to clean up large-omap in the index pool? PG deep_scrub didn't help. I know how to clean in the log pool, but no idea in the index pool :/ It's an octopus deployment 15.2.10. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Nautilus CentOS-7 rpm dependencies
by Andreas Haupt 31 May '21

31 May '21

Dear all, ceph-mgr-dashboard-15.2.13-0.el7.noarch contains three rpm dependencies that cannot be resolved here (not part of CentOS & EPEL 7): python3-cherrypy python3-routes python3-jwt Does anybody know where they are expected to come from? Thanks, Andreas -- | Andreas Haupt | E-Mail: andreas.haupt(a)desy.de | DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt | Platanenallee 6 | Phone: +49/33762/7-7359 | D-15738 Zeuthen | Fax: +49/33762/7-7216

3 2

Cephadm/docker or install from packages
by Stanislav Datskevych 30 May '21

30 May '21

Hi all, I want to ask your opinion which Ceph deploy version is better: using cephadm(docker) or installing from packages? Cephadm brings lots of convenience: easy upgrade (which sometimes stucks but still), easy add new OSDs, ability to make placement policies etc. In contrary I seem to lose some transparency on how Ceph works as well as ability to apply fine-grained control, namely setting up daemon configs myself. Even running ceph-volume lvm list to see which osd uses a particular block devices requires entering a docker container. So which option is better to use in production cluster?

1 0

SSD recommendations for RBD and VM's
by by morphin 30 May '21

30 May '21

Hello. I have virtualization env and I'm looking new SSD for HDD replacement. What are the best Performance / Price SSDs in the market right now? I'm looking 1TB, 512GB, 480GB, 256GB, 240GB. Is there a SSD recommendation list for ceph?

3 3

nomenclature: ceph or cephfs (initramfs-tools)
by Markus Kienast 29 May '21

29 May '21

Hi, I have created initramfs scripts for booting from RBD and CephFS: https://launchpad.net/~trickkiste/+archive/ubuntu/ceph-initramfs https://gitlab.com/trickkiste-at/ceph-initramfs I noticed that the CephFS kernel module seems to be named "ceph" not "cephfs", so I would like to to inquire whether I should also name my initramfs-tools scripts and hooks accordingly - ergo "ceph" (not cephfs). My best regards Markus

1 0

HEALTH_WARN Reduced data availability: 33 pgs inactive
by Jeremy Hansen 29 May '21

29 May '21

I’m trying to understand this situation: ceph health detail HEALTH_WARN Reduced data availability: 33 pgs inactive [WRN] PG_AVAILABILITY: Reduced data availability: 33 pgs inactive pg 1.0 is stuck inactive for 20h, current state unknown, last acting [] pg 2.0 is stuck inactive for 20h, current state unknown, last acting [] pg 2.1 is stuck inactive for 20h, current state unknown, last acting [] pg 2.2 is stuck inactive for 20h, current state unknown, last acting [] pg 2.3 is stuck inactive for 20h, current state unknown, last acting [] pg 2.4 is stuck inactive for 20h, current state unknown, last acting [] pg 2.5 is stuck inactive for 20h, current state unknown, last acting [] pg 2.6 is stuck inactive for 20h, current state unknown, last acting [] pg 2.7 is stuck inactive for 20h, current state unknown, last acting [] pg 2.8 is stuck inactive for 20h, current state unknown, last acting [] pg 2.9 is stuck inactive for 20h, current state unknown, last acting [] pg 2.a is stuck inactive for 20h, current state unknown, last acting [] pg 2.b is stuck inactive for 20h, current state unknown, last acting [] pg 2.c is stuck inactive for 20h, current state unknown, last acting [] pg 2.d is stuck inactive for 20h, current state unknown, last acting [] pg 2.e is stuck inactive for 20h, current state unknown, last acting [] pg 2.f is stuck inactive for 20h, current state unknown, last acting [] pg 2.10 is stuck inactive for 20h, current state unknown, last acting [] pg 2.11 is stuck inactive for 20h, current state unknown, last acting [] pg 2.12 is stuck inactive for 20h, current state unknown, last acting [] pg 2.13 is stuck inactive for 20h, current state unknown, last acting [] pg 2.14 is stuck inactive for 20h, current state unknown, last acting [] pg 2.15 is stuck inactive for 20h, current state unknown, last acting [] pg 2.16 is stuck inactive for 20h, current state unknown, last acting [] pg 2.17 is stuck inactive for 20h, current state unknown, last acting [] pg 2.18 is stuck inactive for 20h, current state unknown, last acting [] pg 2.19 is stuck inactive for 20h, current state unknown, last acting [] pg 2.1a is stuck inactive for 20h, current state unknown, last acting [] pg 2.1b is stuck inactive for 20h, current state unknown, last acting [] pg 2.1c is stuck inactive for 20h, current state unknown, last acting [] pg 2.1d is stuck inactive for 20h, current state unknown, last acting [] pg 2.1e is stuck inactive for 20h, current state unknown, last acting [] pg 2.1f is stuck inactive for 20h, current state unknown, last acting [] [ceph: root@cn01 /]# date Sat May 29 01:28:37 UTC 2021 [ceph: root@cn01 /]# ceph pg dump_stuck inactive PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 2.1f unknown [] -1 [] -1 2.1e unknown [] -1 [] -1 2.1d unknown [] -1 [] -1 2.1c unknown [] -1 [] -1 2.1b unknown [] -1 [] -1 2.1a unknown [] -1 [] -1 2.19 unknown [] -1 [] -1 2.18 unknown [] -1 [] -1 2.17 unknown [] -1 [] -1 2.16 unknown [] -1 [] -1 2.15 unknown [] -1 [] -1 2.14 unknown [] -1 [] -1 2.13 unknown [] -1 [] -1 2.12 unknown [] -1 [] -1 2.11 unknown [] -1 [] -1 2.10 unknown [] -1 [] -1 2.f unknown [] -1 [] -1 2.9 unknown [] -1 [] -1 2.b unknown [] -1 [] -1 2.c unknown [] -1 [] -1 2.e unknown [] -1 [] -1 2.a unknown [] -1 [] -1 2.d unknown [] -1 [] -1 2.8 unknown [] -1 [] -1 2.7 unknown [] -1 [] -1 2.6 unknown [] -1 [] -1 2.5 unknown [] -1 [] -1 2.0 unknown [] -1 [] -1 1.0 unknown [] -1 [] -1 2.3 unknown [] -1 [] -1 2.1 unknown [] -1 [] -1 2.2 unknown [] -1 [] -1 2.4 unknown [] -1 [] -1 ok [ceph: root@cn01 /]# ceph pg 2.4 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1310, in <module> retval = main() File "/usr/bin/ceph", line 1230, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 966, in parse_json_funcsigs raise e File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 963, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) -jeremy

1 0

Remapping OSDs under a PG
by Jeremy Hansen 28 May '21

28 May '21

I’m very new to Ceph so if this question makes no sense, I apologize. Continuing to study but I thought an answer to this question would help me understand Ceph a bit more. Using cephadm, I set up a cluster. Cephadm automatically creates a pool for Ceph metrics. It looks like one of my ssd osd’s was allocated for the PG. I’d like to understand how to remap this PG so it’s not using the SSD OSDs. ceph pg map 1.0 osdmap e205 pg 1.0 (1.0) -> up [28,33,10] acting [28,33,10] OSD 28 is the SSD. Is this possible? Does this make any sense? I’d like to reserve the SSDs for their own pool. Thank you! -jeremy

3 5

cephfs auditing
by Michael Thomas 28 May '21

28 May '21

Is there a way to log or track which cephfs files are being accessed? This would help us in planning where to place certain datasets based on popularity, eg on a EC HDD pool or a replicated SSD pool. I know I can run inotify on the ceph clients, but I was hoping that the MDS would have a way to log this information centrally. --Mike

3 2

XFS on RBD on EC painfully slow
by Reed Dier 28 May '21

28 May '21

Hoping someone may be able to help point out where my bottleneck(s) may be. I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt. To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories. My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal. > 15.41G 79% 4.36MB/s 0:56:09 (xfr#23165, ir-chk=4061/27259) > avg-cpu: %user %nice %system %iowait %steal %idle > 0.17 0.00 1.34 13.23 0.00 85.26 > > Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz %util > rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17 0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39 96.40 Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any bottlenecks doubling back to cephfs. About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm looking to offload to cephfs. Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out. EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here. Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is. Examining the main directory structure it looks like there are 7000 files per directory, about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory. My fstab for this is: > xfs _netdev,noatime 0 0 I tried to increase the read_ahead_kb to 4M from 128K at /sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but that doesn't appear to have had much of an impact. The only thing I can think of that I could possibly try as a change would be to increase the queue depth in the rbdmap up from 128, so thats my next bullet to fire. Attaching xfs_info in case there are any useful nuggets: > meta-data=/dev/rbd0 isize=256 agcount=81, agsize=268435455 blks > = sectsz=512 attr=2, projid32bit=0 > = crc=0 finobt=0, sparse=0, rmapbt=0 > = reflink=0 > data = bsize=4096 blocks=21483470848, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=0 > log =internal log bsize=4096 blocks=32768, version=2 > = sectsz=512 sunit=0 blks, lazy-count=0 > realtime =none extsz=4096 blocks=0, rtextents=0 And rbd-info: > rbd image 'rbd-image-name: > size 85 TiB in 22282240 objects > order 22 (4 MiB objects) > snapshot_count: 0 > id: a09cac2b772af5 > data_pool: rbd-ec82-pool > block_name_prefix: rbd_data.29.a09cac2b772af5 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool > op_features: > flags: > create_timestamp: Mon Apr 12 18:44:38 2021 > access_timestamp: Mon Apr 12 18:44:38 2021 > modify_timestamp: Mon Apr 12 18:44:38 2021 Any other ideas or hints are greatly appreciated. Thanks, Reed

4 4

Re: XFS on RBD on EC painfully slow
by Reed Dier 28 May '21

28 May '21

I guess I should probably have been more clear, this is one pool of many, so the other OSDs aren't idle. So I don't necessarily think that the PG bump would be the worst thing to try, but its definitely not as bad as I may have made it sound. Thanks, Reed > On May 27, 2021, at 11:37 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > That gives you a PG ratio of …. 5.3 ??? > > Run `ceph osd df` ; I wouldn’t be surprised if some of your drives have 0 PGs on them, for sure I would suspect that they aren’t even at all. > > There are bottlenecks in the PG code, and in the OSD code — one reason why with NVMe clusters it’s common to split each drive into at least 2 OSDs. With spinners you don’t want to do that, but you get the idea. > > The pg autoscaler is usually out of its Vulcan mind. 512 would give you a ratio of just 21. > > Prior to 12.2.1 conventional wisdom was a PG ratio of 100-200 on spinners. > > 2048 PGs would give you a ratio of 85, which current (retconned) guidance would call good. I’d probably go to 4096 but 2048 would be way better than 128. > > I strongly suspect that PG splitting would still get you done faster than the way it is, esp. if you’re running BlueStore OSDs. > > Try bumping pg_num up to say 262 and see how bad it is, and if when pgp_num catches up if your ingest rate isn’t a bit higher than it was before. > >> EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here. >> Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is. > >

1 0

Messed up placement of MDS
by mabi 28 May '21

28 May '21

Hello, I am trying to place the two MDS daemons for CephFS on dedicated nodes. For that purpose I tried out a few different "cephadm orch apply ..." commands with a label but at the end it looks like I messed up with the placement as I now have two mds service_types as you can see below: # ceph orch ls --service-type mds --export service_type: mds service_id: ceph1fs service_name: mds.ceph1fs placement: count: 2 hosts: - ceph1g - ceph1a --- service_type: mds service_id: label:mds service_name: mds.label:mds placement: count: 2 This second entry at the bottom seems totally wrong and I would like to remove it but I haven't found how to remove it totally. Any ideas? Ideally I just want to place two MDS daemons on node ceph1a and ceph1g. Regards, Mabi

1 1

Ceph osd will not start.
by Peter Childs 27 May '21

27 May '21

I'm attempting to get get ceph up and running, and currently feel like I'm going around in circles. I'm attempting to use cephadm and Pacific, currently on debian buster, mostly because centos7 ain't supported any more and cenotos8 ain't support by some of my hardware. Anyway I have a few nodes with 59x 7.2TB disks but for some reason the osd daemons don't start, the disks get formatted and the osd are created but the daemons never come up. They are probably the wrong spec for ceph (48gb of memory and only 4 cores) but I was expecting them to start and be either dirt slow or crash later, anyway I've got upto 30 of them, so I was hoping on getting at least get 6PB of raw storage out of them. As yet I've not spotted any helpful error messages. This is for a archive / slow ceph cluster so I'm not expecting speed. Thanks in advance. Peter.

4 8

rebalancing after node more
by Rok Jaklič 27 May '21

27 May '21

Hi, I have removed one node, but now ceph seems to stuck in: Degraded data redundancy: 67/2393 objects degraded (2.800%), 12 pgs degraded, 12 pgs undersized How to "force" rebalancing? Or should I just wait a little bit more? Kind regards, rok

2 5

cephadm: How to replace failed HDD where DB is on SSD
by Kai Stian Olstad 27 May '21

27 May '21

Hi The server run 15.2.9 and has 15 HDD and 3 SSD. The OSDs was created with this YAML file hdd.yml -------- service_type: osd service_id: hdd placement: host_pattern: 'pech-hd-*' data_devices: rotational: 1 db_devices: rotational: 0 The result was that the 3 SSD is added to 1 VG with 15 LV on it. # vgs | egrep "VG|dbs" VG #PV #LV #SN Attr VSize VFree ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n- <5.24t 48.00m One of the osd failed and I run rm with replace # ceph orch osd rm 178 --replace and the result is # ceph osd tree | grep "ID|destroyed" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 178 hdd 12.82390 osd.178 destroyed 0 1.00000 But I'm not able to replace the disk with the same YAML file as shown above. # ceph orch apply osd -i hdd.yml --dry-run ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ I guess this is the wrong way to do it, but I can't find the answer in the documentation. So how can I replace this failed disk in Cephadm? -- Kai Stian Olstad

3 10

How to add back stray OSD daemon after node re-installation
by mabi 27 May '21

27 May '21

Hello, I have by mistake re-installed the OS of an OSD node of my Octopus cluster (managed by cephadm). Luckily the OSD data is on a separate disk and did not get affected by the re-install. Now I have the following state: health: HEALTH_WARN 1 stray daemon(s) not managed by cephadm 1 osds down 1 host (1 osds) down To fix that I tried to run: # ceph orch daemon add osd ceph1f:/dev/sda Created no osd(s) on host ceph1f; already created? That did not work, so I tried: # ceph cephadm osd activate ceph1f no valid command found; 10 closest matches: ... Error EINVAL: invalid command Did not work either. So I wanted to ask how can I "adopt" back an OSD disk to my cluster? Thanks for your help. Regards, Mabi

2 10

MDS stuck in up:stopping state
by Martin Rasmus Lundquist Hansen 27 May '21

27 May '21

After scaling the number of MDS daemons down, we now have a daemon stuck in the "up:stopping" state. The documentation says it can take several minutes to stop the daemon, but it has been stuck in this state for almost a full day. According to the "ceph fs status" output attached below, it still holds information about 2 inodes, which we assume is the reason why it cannot stop completely. Does anyone know what we can do to finally stop it? cephfs - 71 clients ====== RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-mon-01 Reqs: 0 /s 15.7M 15.4M 1 active ceph-mon-02 Reqs: 48 /s 19.7M 17.1M 2 stopping ceph-mon-03 0 2 POOL TYPE USED AVAIL cephfs_metadata metadata 652G 185T cephfs_data data 1637T 539T STANDBY MDS ceph-mon-03-mds-2 MDS version: ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)

3 7

cephfs:: store files on different pools?
by Adrian Sevcenco 27 May '21

27 May '21

Hi! is is (technically) possible to instruct cephfs to store files < 1Mib on a (replicate) pool and the others files on another (ec) pool? And even more, is it possible to take the same kind of decision on the path of the file? (let's say that critical files with names like r"/critical_path/critical_.*" i want them in a 6x replication ssd pool) Thank you! Adrian

2 1

Re: cephadm: How to replace failed HDD where DB is on SSD
by Eugen Block 27 May '21

27 May '21

Could you share the output of lsblk -o name,rota,size,type from the affected osd node? My spec file is for a tiny lab cluster, in your case the db drive size should be something like '5T:6T' to specify a range. How large are the HDDs? Also maybe you should use the option 'filter_logic: AND', but I'm not sure if that's already the default, I remember that there were issues in Nautilus because the default was OR. I tried this just recently with a version similar to this, I believe it was 15.2.8 and it worked for me, but again, it's just a tiny virtual lab cluster. Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>: > On 26.05.2021 11:16, Eugen Block wrote: >> Yes, the LVs are not removed automatically, you need to free up the >> VG, there are a couple of ways to do so, for example remotely: >> >> pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force >> >> or directly on the host with: >> >> pacific1:~ # cephadm ceph-volume lvm zap --destroy /dev/<CEPH_VG>/<DB_LV> > > Thanks, > > I used the cephadm command and deleted the LV and the VG now has free space > > # vgs | egrep "VG|dbs" > VG #PV #LV #SN > Attr VSize VFree > ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 14 0 > wz--n- <5.24t 357.74g > > But it doesn't seams to be able to use it, because it can find anyting > > # ceph orch apply osd -i hdd.yml --dry-run > ################ > OSDSPEC PREVIEWS > ################ > +---------+------+-------------+----------+----+-----+ > |SERVICE |NAME |HOST |DATA |DB |WAL | > +---------+------+-------------+----------+----+-----+ > +---------+------+-------------+----------+----+-----+ > > I tried adding size as you have in your configuration > db_devices: > rotational: 0 > size: '30G:' > > Still it was unable to create the OSD. > > If I removed the : so it is 30GB exact size, it did find the disk, > but DB is not placed on a SSD since I do not have one with 30 GB > exact size > ################ > OSDSPEC PREVIEWS > ################ > +---------+------+-------------+----------+----+-----+ > |SERVICE |NAME |HOST |DATA |DB |WAL | > +---------+------+-------------+----------+----+-----+ > |osd |hdd |pech-hd-7 |/dev/sdt |- |- | > +---------+------+-------------+----------+----+-----+ > > > To me I looks like Cephadm can't use/find the free space on the VG > and use that as a new LV for the OSD. > > > -- > Kai Stian Olstad > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

best practice balance mode in HAproxy in front of RGW?
by Boris Behrens 27 May '21

27 May '21

Hello togehter, is there any best practive on the balance mode when I have a HAproxy in front of my rgw_frontend? currently we use "balance leastconn". Cheers Boris

2 4

Python lib usage access permissions
by Szabo, Istvan (Agoda) 27 May '21

27 May '21

Hi, Is there a way to be able to manage specific pools with the python lib without admin keyring? Not sure why it only works with admin keyring but not with the client keyring :/ Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

v15.2.13 Octopus released
by David Galloway 26 May '21

26 May '21

We're happy to announce the 13th backport release in the Octopus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-13-octopus-released Notable Changes --------------- * RADOS: Ability to dynamically adjust trimming rate in the monitor and several other bug fixes. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-15.2.13.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: c44bc49e7a57a87d84dfff2a077a2058aa2172e2

1 0

cephfs vs rbd vs rgw
by Jorge Garcia 26 May '21

26 May '21

This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question: Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts: cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance. rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS. rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface? I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past. Thanks for any help you can provide! Jorge

5 4

summarized radosgw size_kb_actual vs pool stored value doesn't add up
by Boris Behrens 26 May '21

26 May '21

Hi, I am still searching for a reason why these two values differ so much. I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller. This was the state two days ago: > > [root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc > 175977343264 > > [root@s3db1 ~]# rados df > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR > ... > eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0 0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B > ... > total_objects 135866676 > > [root@s3db1 ~]# ceph df... > eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB 90.32 27 TiB And this is todays state: > > [root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc > 177144806812 > > [root@s3db1 ~]# rados df > ... > eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770 > ... > total_objects 121261889 > > [root@s3db1 ~]# ceph df > ... > eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB 92.59 21 TiB I would love to free up the missing 80TB :) Any suggestions? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 4

Pacific: _admin label does not distribute admin keyring
by Eugen Block 26 May '21

26 May '21

Hi, I'm currently exploring Pacific and the "_admin" label doesn't seem to work as expected. pacific1:~ # ceph -v ceph version 16.2.3-26-g422932e923 (422932e923eb429b9e16c352a663968f4b6f0a52) pacific (stable) According to the docs [1] the "_admin" label should instruct cephadm to distribute a ceph.conf and the admin keyring to the labeled host: > _admin: Distribute client.admin and ceph.conf to this host. > > By default, an _admin label is applied to the first host in the > cluster (where bootstrap was originally run), and the client.admin > key is set to be distributed to that host via the ceph orch > client-keyring ... function. Adding this label to additional hosts > will normally cause cephadm to deploy config and keyring files in > /etc/ceph. I can not confirm this, neither has the bootstrap node (pacific1) the "_admin" label (but he has the ceph.conf and admin keyring) nor does pacific2 have the admin keyring. Here's my host ls: pacific1:~ # ceph orch host ls HOST ADDR LABELS STATUS pacific1 192.168.124.183 mon pacific2 192.168.124.49 _admin mon pacific5 192.168.124.63 mon pacific1:~ # ssh pacific2 find /etc/ceph /etc/ceph /etc/ceph/rbdmap pacific1:~ # find /etc/ceph/ /etc/ceph/ /etc/ceph/ceph.pub /etc/ceph/ceph.conf /etc/ceph/rbdmap /etc/ceph/ceph.client.admin.keyring I searched a bit on tracker.ceph.com but couldn't find an existing issue. Is this worth a bug report? Regards, Eugen [1] https://docs.ceph.com/en/latest/cephadm/host-management/#special-host-labels

1 0

Ceph Pacific mon is not starting after host reboot
by Adrian Nicolae 25 May '21

25 May '21

Hi guys, I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put it in production on a 1PB+ storage cluster with rgw-only access. I noticed a weird issue with my mons : - if I reboot a mon host, the ceph-mon container is not starting after reboot - I can see with 'ceph orch ps' the following output : mon.node01 node01 running (20h) 4m ago 20h 16.2.4 8d91d370c2b8 0a2e86af94b2 mon.node02 node02 running (115m) 12s ago 115m 16.2.4 8d91d370c2b8 51f4885a1b06 mon.node03 node03 stopped 4m ago 19h <unknown> <unknown> <unknown> (where node03 is the host which was rebooted). - I tried to start the mon container manually on node03 with '/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and I've got the following output : debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 3314933069573799936, adjusting msgr requires debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112, adjusting msgr requires debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112, adjusting msgr requires debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112, adjusting msgr requires cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164) 36380 : cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used, 651 GiB / 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1 mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format 0 -> 3 debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after 0.000000000s debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing) e5 my rank is now 1 (was -1) debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing) e6 removed from monmap, suicide. root@node03:/home/adrian# systemctl status ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service ● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph mon.node03 for c2d41ac4-baf5-11eb-865d-2dc838a337a3 Loaded: loaded (/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago Process: 1176 ExecStart=/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited, status=0/SUCCESS) Process: 1855 ExecStop=/usr/bin/docker stop ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE) Process: 1861 ExecStopPost=/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited, status=0/SUCCESS) Main PID: 1176 (code=exited, status=0/SUCCESS) The only fix I could find was to redeploy the mon with : ceph orch daemon rm mon.node03 --force ceph orch daemon add mon node03 However, even if it's working after redeploy, it's not giving me a lot of trust to use it in a production environment having an issue like that. I could reproduce it with 2 different mons so it's not just an exception. My setup is based on Ubuntu 20.04 and docker instead of podman : root@node01:~# docker -v Docker version 20.10.6, build 370c289 Do you know a workaround for this issue or is this a known bug ? I noticed that there are some other complaints with the same behaviour in Octopus as well and the solution at that time was to delete the /var/lib/ceph/mon folder . Thanks.

5 10

Very uneven OSD utilization
by Sergei Genchev 25 May '21

25 May '21

Hello, I am running a nautilus cluster with 5 OSD nodes/90 disks that is exclusively used for S3. My disks are identical, but utilization ranges from 9% to 82%, and I am starting to get backfill_toofull errors even though I have only used 150TB out of 650TB of data. - Other than manually crush reweighting OSDs, is there any other option for me ? - what would cause this uneven distribution? Is there some documentation on how to track down what's going on? output of 'ceph osd df" is at https://pastebin.com/17HWFR12 Thank you!

2 2

rbd cp versus deep cp?
by Jan Kasprzak 25 May '21

25 May '21

Hello, Ceph users, what is the difference between "rbd cp" and "rbd deep cp"? What I need to do is to make a copy of the rbd volume one of our users inadveredly resized to a too big size, shrink the copied image to the expected size, verify that everything is OK, and then delete the original image. Would this work with rbd cp? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

OSD and RBD on same node?
by Tuffli, Chuck 25 May '21

25 May '21

I found a warning in Red Hat documentation regarding OSD and RBD: Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and OSD nodes. Running kernel clients and kernel server daemons on the same node can lead to kernel deadlocks. [1] It was hard to tell from the documentation if this is still true as some Rook documentation [2] mentioned a solution. Is having an RBD on the same node as the OSD's still a no-no or is it OK with the conditions listed in the Rook docs (i.e. patched kernel + no XFS)? TIA! --chuck [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/i… [2] https://rook.io/docs/rook/v1.6/ceph-common-issues.html#a-worker-node-using-…

3 2

DocuBetter Meeting 1AM UTC Thursday 27 May 2021
by John Zachary Dover 24 May '21

24 May '21

A DocuBetter meeting is scheduled for later this week at 11AM AEST Thursday, which is 6PM PDT Wednesday and 9PM EDT Wednesday (and 1AM UTC). It's a pretty inconveniently scheduled meeting. Last month I sent an email reporting that this monthly meeting, on account of its inconvenient time, is not much attended. That email provoked good, helpful correspondence, so I'm doing it again. Unless I get responses to this email thread that request that I hold the meeting this week, I'm not going to hold it. This email is a sincere request for documentation complaints. If anything about the documentation irritates you, now's the time to tell me. If anything about the documentation is incomplete or incorrect, I'm the guy you should tell. You don't have to attend the DocuBetter Meetings to get changes into the documentation, you just have to ask me and explain clearly what needs to be changed. The documentation initiatives that were underway last month remain underway sedulously, if not apace. I am cleaning the syntax of the entire documentation suite (a multi-month project that is as tedious as it sounds... I am currently in the middle of the cephadm documentation). I am also planning to consolidate the Intro Guide and part of the Developer Guide, because there is some overlap in those sections. I've also begun indexing the Ceph-related videos on Youtube. This means that I watch the videos with a pencil in my hand, and I write down short summaries of what's being discussed along with the number of minutes and seconds that have passed since the video began. Over the past two weeks I have found myself with command of the context and understanding necessary to appreciate Sage's one-hour-and-twenty-seven-minute marathon "Intro to Ceph" talk from June 2019 (https://www.youtube.com/watch?v=PmLPbrf-x9g) I'm not sure that a video six minutes longer than "Toy Story" can be considered an intro, but it is an impressive rhetorical performance by Sage, who speaks authoritatively and concisely on every subject he touches as he provides a tour of all the major components of Ceph. Anyway: If someone responds to this email and says that their lives would be improved if I hold the DocuBetter meeting, then I will hold the meeting. However, if by twelve hours before the time of the meeting no one has responded, there will be no DocuBetter meeting. Remember: even if no one wants to have the meeting this week, this does not mean that you can't get changes into the docs. Write to me anytime with your complaints. No complaint is too crude, no irritation will be summarily dismissed. That's it. Here are the links relevant to DocuBetter Meetings: Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

upmap+assimilate-conf clarification
by Szabo, Istvan (Agoda) 24 May '21

24 May '21

Hi, I've successfully updated my luminous lab environment to nautilus so next week I'll give a try to the prod env but 2 things same up in the description: 1. Upmap: I've never used this before don't know how I couldn't see it because it is quite cool feature. So ceph says let it in upmap, my question is like with the ceph-pg autoscaler module not the best thing to use in 'on' mode, better in warn, so is it good to use in upmap mode let's say in ceph world "safe"? 2. Regarding this assimilate-conf. It's not clear for me what is it actually? Doc says: "This is also a good time to fully transition any config options in ceph.conf into the cluster's configuration database. On each host, you can use the following command to import any option into the monitors with ceph config assimilate-conf -i /etc/ceph/ceph.conf". What does this mean? If I have different configs on different servers and I run it on the servers it will merge it together all the configs? Or what I can do with this feature? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Does dynamic resharding block I/Os by design?
by Satoru Takeuchi 24 May '21

24 May '21

Hi, I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not? I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30. I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding. https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/ In addition, I read the following Red Hat's blog post. https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance… > You do not need to stop reading or writing objects to the bucket while resharding is happening. It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not. Thanks, Satoru

2 4

How to organize data in S3
by Michal Strnad 24 May '21

24 May '21

Hi all, We need to store millions of files using S3 protocol in Ceph (version Nautilus), but have projects where isn't appropriate or possible to create a lot of S3 accounts. Is it better to have multiple S3 buckets or one bucket with sub folders? For example AWS service from Amazon allows you to create up to 100 buckets in each of your AWS cloud accounts. You can request more buckets, up to a maximum quota of 1,000, by submitting a service limit increase. There is no limit on the number of objects you can store in a bucket, but in the Ceph we run into a problem with listing and resharding with a millions of files in one bucket. Thank you Michal

4 5

Recommendations on problem with PG
by Gabriel Medve 23 May '21

23 May '21

Hi, We have a problem with a PG that was inconsistent, currently the PG in our cluster have 3 copies. It was not possible for us to repair this pg with "ceph pg repair" (This PG is in osd 14,1,2) so we deleted some of the copies of osd 14 with the following command. ceph-objectstore-tool --data-path /var/lib/ceph/osd.14/ --pgid 22.f --op remove --force This caused an automatic attempt to create the missing copy entering the backfilling state, but when doing this it crashed osd 1 and 2 and threw the IOPS to 0, freezing the cluster. Is there any way to remove this entire pg or try to recreate the missing copy or ignore it completely? It causes instability in the cluster. Thank you, I await comments -- Untitled Document ------------------------------------------------------------------------ Gabriel I. Medve

2 2

One mds daemon damaged, filesystem is offline. How to recover?
by Sagara Wijetunga 23 May '21

23 May '21

Hi all An accidental power failure happened. That resulted CephFS offline and cannot be mounted. I have 3 MDS daemons but it complains "1 mds daemon damaged". It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get it repaired. How do I repair the damaged MDS and bring the CephFS up/online? Details are included below. Many thanks in advance. Sagara # ceph -s cluster: id: abc... health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 4 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum a,b,c (age 107s) mgr: a(active, since 22m), standbys: b, c mds: cephfs:0/1 3 up:standby, 1 damaged osd: 3 osds: 3 up (since 96s), 3 in (since 96s) data: pools: 3 pools, 192 pgs objects: 281.05k objects, 327 GiB usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail pgs: 191 active+clean 1 active+clean+inconsistent # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged; 4 scrub errors; Possible data damage: 1 pg inconsistent FS_DEGRADED 1 filesystem is degraded fs cephfs is degraded MDS_ALL_DOWN 1 filesystem is offline fs cephfs is offline because no MDS is active for it. MDS_DAMAGE 1 mds daemon damaged fs cephfs mds.0 is damaged OSD_SCRUB_ERRORS 4 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 2.44 is active+clean+inconsistent, acting [0,2,1] # ceph osd lspools 2 cephfs_metadata 3 cephfs_data 4 rbd # ceph pg repair 2.44 # ceph -w 2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid 2:22efaf6a:::200.00006048:head : candidate size 1441792 info size 1555896 mismatch 2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid 2:22efaf6a:::200.00006048:head : failed to pick suitable object info 2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44 2:22efaf6a:::200.00006048:head : on disk size (1540096) does not match object info size (1555896) adjusted for ondisk to (1555896) 2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed --- End of detail ---

3 15

Re: One mds daemon damaged, filesystem is offline. How to recover?
by Eugen Block 22 May '21

22 May '21

Oh these are filestore OSDs? I didn't expect that. > 2021-05-20 23:56:52.223 2227200 0 mds.0.journaler.mdlog(ro) > _finish_read got less than expected (1555896) > 2021-05-20 23:56:52.223 2229180 0 mds.0.log _replay journaler got > error -22, aborting > 2021-05-20 23:56:52.223 2229180 -1 log_channel(cluster) log [ERR] : > Error loading MDS rank 0: (22) Invalid argument This confirms that the MDS is not starting because of the object size. I assume there was some recovery going on when this happened? The OSD uptime from your status was quite short. I'm not really sure what to do next, I found an old thread [4] with almost the same error, only that the user ended up truncating the objects because the expected size was smaller than the object info, in your case they're the other way around. In any case I would backup all three copies of that object including md5sum before chaning anything. [4] https://www.spinics.net/lists/ceph-devel/msg16516.html Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>: > Here are the physical file sizes of the "200.00006048*": > OSD.0:-rw-r--r-- 1 ceph ceph 1540096 May 20 22:47 > /var/lib/ceph/osd/ceph-0/current/2.44_head/200.00006048__head_56F5F744__ > > OSD.1:-rw-r--r-- 1 ceph ceph 1540096 May 20 22:47 > /var/lib/ceph/osd/ceph-1/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2 > > OSD2:-rw-r--r-- 1 ceph ceph 1441792 May 20 22:47 > /var/lib/ceph/osd/ceph-2/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2 > > Sagara > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph orch status hangs forever
by ManuParra 22 May '21

22 May '21

Hi Eugen, this is the output: ceph mgr module ls { "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator", "pg_autoscaler", "progress", "rbd_support", "status", "telemetry", "volumes" ], "enabled_modules": [ "cephadm", "dashboard", "diskprediction_local", "iostat", "prometheus", "restful" ], "disabled_modules": [ ... } As you see balancer/crash/… are in section always_on. I checked it on all of the 3 monitor nodes with the same output. Then, checking disabled_modules I’ve seen many modules that could help to track some more information (logs) on our problem, like: - alerts - insights - test_orchestrator - and other… But we are not sure if we can enable some of them. Now all the logs we have from Ceph are not showing errors. Would it help to see more logs to enable some of those modules? On the other hand, as for what you indicate us of the commands that hang, we see that the containers that launch it remain executing, since they are waiting to finish. So you can see here the list of tests that remain running (hung): af13bda77a1a 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph osd status" 23 hours ago Up 23 hours wizardly_leavitt 5b5c760454c7 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph telemetry stat…" 24 hours ago Up 24 hours intelligent_bardeen a98e6061489d 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph service dump" 24 hours ago Up 24 hours romantic_mendel 66c943a032f8 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph service status" 24 hours ago Up 24 hours happy_shannon 7e18899dffc5 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph crash stat" 24 hours ago Up 24 hours xenodochial_germain 8268082e753b 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph crash ls" 24 hours ago Up 24 hours stoic_volhard fc5c434a4e23 172.16.3.146:4000/ceph/ceph:v15.2.9 "ceph balancer status" 24 hours ago Up 24 hours epic_mendel So the containers will have to be eliminated. As for the logs of these containers nothing appears inside the container (docker logs xxxx), only, when you kill it you can see (--verbose): [ceph: root@spsrc-mon-1 /]# ceph —verbose pg stat …. validate_command: pg stat better match: 2.5 > 0: pg stat bestcmds_sorted: [{'flags': 8, 'help': 'show placement group status.', 'module': 'pg', 'perm': 'r', 'sig': [argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=pg), argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=stat)]}] Submitting command: {'prefix': 'pg stat', 'target': ('mon-mgr', '')} submit ['{"prefix": "pg stat", "target": ["mon-mgr", ""]}'] to mon-mgr [hung forever …] Kind regards, Manu.

3 3

Force processing of num_strays in mds
by Mark Schouten 21 May '21

21 May '21

Hi, I have a 12.2.13 I want to go and upgrade. However, there are a whole bunch of stray files/inodes(?) which I would want to have processed. Also because I get a lot of 'No space left on device' messages. I started a 'find . -ls' in the root of the CephFS filesystem, but that causes overload and takes a lot of time, while not neccesarily fixing the num_strays. How do I force the mds'es to process those strays so that clients do not get 'incorrect' errors? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

2 1

mgr+Prometheus, grafana, consul
by Jeremy Austin 21 May '21

21 May '21

I recently configured Prometheus to scrape mgr /metrics and add Grafana dashboards. All daemons currently at 15.2.11 I use Hashicorp consul to advertise the active mgr in DNS, and Prometheus points at a single DNS target. (Is anyone else using this method, or just statically pointing Prometheus at all potentially active managers?) All was working fine initially, and it's *mostly* still working fine. For the first couple of days, all went well, and then a few rate metrics stopped meaningfully increasing — essentially pegged at zero, which is implausible in a healthy cluster. Some cluster maintenance was occurring such as outing and recreating some OSDs, so I have a baseline for throughput and recovery. Metric graphs that stopped functioning: Throughput: ceph_osd_op_r_out_bytes, ceph_osd_op_w_in_bytes, ceph_osd_op_rw_in_bytes Recovery: ceph_osd_recovery_ops I can see that Grafana output is using this method of converting the counters to rates: sum(irate(ceph_osd_recovery_ops{job="$job"}[$interval])) The underlying counters appear to be sane, and reading the raw values from prometheus is also valid, so I'm guessing some failure either of the irate or sum functions? By inspection in Grafana, the queries return correct timestamps with zero values, so that leaves us with "sum(irate)" as the likely source of the problem. Does anyone have experience with this? I admit it is possibly tangential to ceph itself, but as the Prometheus/grafana integration is more or less supported, I thought I'd try here first among active mgr/Prometheus users. -- Jeremy Austin jhaustin(a)gmail.com

1 0

OSD's still UP after power loss
by by morphin 21 May '21

21 May '21

Hello I have a weird problem on 3 node cluster. "Nautilus 14.2.9" When I try power failure OSD's are not marking as DOWN and MDS do not respond anymore. If I manually set osd down then MDS becomes active again. BTW: Only 2 node has OSD's. Third node is only for MON. I've set mon_osd_down_out_interval = 0.3 in ceph.conf at global section and restart all MON's but when I check it with "ceph daemon mon.ID config show" I see mon_osd_down_out_interval: "600". I didn't get it why its still "600" and honestly I don't know even it has any effect on my problem. Where should I check?

3 6

question regarding markers in radosgw
by Boris Behrens 21 May '21

21 May '21

Hello everybody, It seems that I have a metric ton of orphan objects in my s3 cluster. They look like this: $ rados -p eu-central-1.rgw.buckets.data stat ff7a8b0c-07e6-463a-861b-78f0adeba8ad.811806.9_1063978/features/2018-02-23.json eu-central-1.rgw.buckets.data/ff7a8b0c-07e6-463a-861b-78f0adeba8ad.811806.9_1063978/features/2018-02-23.json mtime 2018-02-23 20:59:32.000000, size 608 Now I would imagine that 811806.9 is the marker, but when I do a simple radosgw-admin bucket stats | grep -F 811806.9 I get back no results. Can I jsut delete these files? And if I can delete these files, how can I delete them fast? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

ceph osd df size shows wrong, smaller number
by Rok Jaklič 21 May '21

21 May '21

Hi, why would ceph osd df show in SIZE field smaller number than there is: 85 hdd 0.89999 1.00000 100 GiB 96 GiB 95 GiB 289 KiB 952 MiB 4.3 GiB 95.68 3.37 10 up instead of 100GiB there should be 5.5TiB. Kind regards, Rok

3 5

Re: ceph df: pool stored vs bytes_used -- raw or not?
by Konstantin Shalygin 21 May '21

21 May '21

Dan, Igor Seems this wasn't backported? Get stored == used on luminous->Nautilus 14.2.21. What solution is? Find OSD's with zero bytes reported, drain->deploy it? Thanks, k Sent from my iPhone

2 2

Re: ceph orch status hangs forever
by ManuParra 21 May '21

21 May '21

Hi Eugen thank you very much for your reply. I'm Manuel, a colleague of Sebastián. I complete what you ask us. We have checked more ceph commands, not only ceph crash and ceph org and many other commands are equally hung: [spsrc-mon-1 ~]# cephadm shell -- ceph pg stat hangs forever [spsrc-mon-1 ~]# cephadm shell -- ceph status Works [spsrc-mon-1 ~]# cephadm shell -- ceph progress hangs forever [spsrc-mon-1 ~]# cephadm shell -- ceph balancer status hangs forever [spsrc-mon-1 ~]# cephadm shell -- ceph crash ls hangs forever [spsrc-mon-1 ~]# cephadm shell -- ceph crash stat hangs forever [spsrc-mon-1 ~]# cephadm shell -- ceph telemetry status hangs forever We have checked the call made from the container by checking DEBUG logs and I see that it is correct, in some commands work but others hang: 2021-05-20 09:56:02,903 DEBUG Running command (timeout=None): /bin/docker run --rm --ipc=host --net=host --privileged --group-add=disk -e CONTAINER_IMAGE=172.16.3.146:4000/ceph/ceph:v15.2.9 -e NODE_NAME=spsrc-mon-1 -v /var/run/ceph/3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c:/var/run/ceph:z -v /var/log/ceph/3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c:/var/log/ceph:z -v /var/lib/ceph/3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c/mon.spsrc-mon-1/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph 172.16.3.146:4000/ceph/ceph:v15.2.9 pg stat We have 3 monitor nodes and these are the containers that are running (on all monitor nodes): acf8870fc788 172.16.3.146:4000/ceph/ceph:v15.2.9 "/usr/bin/ceph-mds -…" 7 days ago Up 7 days ceph-3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c-mds.manila.spsrc-mon-1.gpulzs cfac86f29db4 172.16.3.146:4000/ceph/ceph:v15.2.9 "/usr/bin/ceph-mon -…" 7 days ago Up 7 days ceph-3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c-mon.spsrc-mon-1 4e6e600fa915 172.16.3.146:4000/ceph/ceph:v15.2.9 "/usr/bin/ceph-crash…" 7 days ago Up 7 days ceph-3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c-crash.spsrc-mon-1 dae36c48568e 172.16.3.146:4000/ceph/ceph:v15.2.9 "/usr/bin/ceph-mgr -…" 7 days ago Up 7 days ceph-3cdbf59a-a74b-11ea-93cc-f0d4e2e6643c-mgr.spsrc-mon-1.eziiam All with running status in all the 3 monitor nodes. As you see in this monitor, we have MDS, MON, CRASH and MGR. Any ideas what we can check?. Best regards, Manu

3 3

Fw: Welcome to the "ceph-users" mailing list
by 274456702＠qq.com 21 May '21

21 May '21

274456702(a)qq.com From: ceph-users-request Date: 2021-05-21 13:55 To: 274456702(a)qq.com Subject: Welcome to the "ceph-users" mailing list Welcome to the "ceph-users" mailing list! To post to this list, send your email to: ceph-users(a)ceph.io You can unsubscribe or make adjustments to your options via email by sending a message to: ceph-users-request(a)ceph.io with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

1 0

Application for mirror.csclub.uwaterloo.ca as an official mirror
by Zachary Seguin 21 May '21

21 May '21

Hello, I am contacting you on behalf of the Computer Science Club of the University of Waterloo (https://csclub.uwaterloo.ca) to add our mirror (https://mirror.csclub.uwaterloo.ca) as an official mirror of the Ceph project. Our mirror is located at the University of Waterloo in Waterloo, Ontario, Canada. The official contact for the mirror is systems-committee(a)csclub.uwaterloo.ca. Following the instructions located at https://ceph.io/get/, I have confirmed that our mirror meets the requirements: * 1Gbit connection or more Our mirror is connected to the University's network at 10Gbps, but is restricted to the internet at 1Gbps on each of the University's 3 10Gbps internet links. * Native IPv4 and IPv6 Our mirror is accessible over native IPv4 and IPv6. mirror.csclub.uwaterloo.ca -------------------------- 129.97.134.71 2620:101:f000:4901:c5c::f:1055 * HTTP access * rsync access Our mirror offers HTTP, HTTPS, FTP and RSYNC access. HTTP: http://mirror.csclub.uwaterloo.ca/ceph/ HTTPS: https://mirror.csclub.uwaterloo.ca/ceph/ FTP: ftp://mirror.csclub.uwaterloo.ca/ceph/ RSYNC: rsync://mirror.csclub.uwaterloo.ca/ceph/ * 2TB of storage or more We have plenty of space to mirror ceph. * Monitoring of the mirror/source We have automated monitoring tracking mirror availability. * Logs Logs from HTTP/HTTPS and RSYNC requests are being stored. We are currently configured to sync from the source repository every 3 hours. The initial sync of the repository has been completed. Please let me know if there is any additional information required and/or any questions. Thanks, Zachary Seguin Systems Committee Computer Science Club | University of Waterloo syscom(a)csclub.uwaterloo.ca

1 0

MDS Stuck in Replay Loop (Segfault) after subvolume creation
by Carsten Feuls 20 May '21

20 May '21

Hello, i want to test something with cephfs subvolume an how to mount it and set quota. after some "ceph fs" commands I got an E-Mail from Prometheus that the cluster is in "Health Warn". The Error was that every MDS crash with a Segfault. Following Some Information of my cluster. The cluster is running via podman. ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable) The commands I habe used to setup my subvolume. ceph fs subvolumegroup create cephfs test-mount # Size 10G ceph fs subvolume create cephfs test-mount-volume test-mount --size=1073741824 ceph fs subvolume info cephfs test-mount-volume test-mount # Failed ceph fs subvolume authorize cephfs test-mount-volume test-mount-client test-mount / # Success ceph fs subvolume authorize cephfs test-mount-volume test-mount-client test-mount Now i got from Prometheus a Health Warn Alert. The reason is MDS stuck in "replay" loop. No MDS come up anymore All MDS crashes with the following Error Message. # MDS Error Message replayed ESubtreeMap at 12471325829 subtree root 0x1 is not mine in cache (it's -2,-2) *** Caught signal (Segmentation fault) ** The journal looks like OK. cephfs-journal-tool --rank=cephfs:all journal inspect Overall journal integrity: OK I took a backup from the journal via. cephfs-journal-tool --rank=cephfs:all journal export backup.bin and an export from the cephfs_metadata pool. rados -p cephfs_metadata export cephfs_metadata_backup A short Output of the events. cephfs-journal-tool --rank=cephfs:all event get list https://pastebin.com/jUDTQL2U[1] I would take the following actions to recover the mds failure: cephfs-journal-tool --rank=cephfs:all event recover_dentries summary cephfs-journal-tool --rank=cephfs:all journal reset cephfs-table-tool --rank=cephfs:all reset session Any suggestions how to fix this? Thanks -- Carsten Feuls -------- [1] https://pastebin.com/jUDTQL2U

1 0

Stray hosts and daemons
by Vladimir Brik 20 May '21

20 May '21

I am not sure how to interpret CEPHADM_STRAY_HOST and CEPHADM_STRAY_DAEMON warnings. They seem to be inconsistent. I converted my cluster to be managed by cephadm by adopting mon and all other daemons, and they show up in ceph orch ps, but ceph health says mons are stray: [WRN] CEPHADM_STRAY_HOST: 6 stray host(s) with 6 daemon(s) not managed by cephadm stray host ceph-6.icecube.wisc.edu has 1 stray daemons: ['mon.ceph-6'] ... At the same time, mon.ceph-6 is not mentioned in the CEPHADM_STRAY_DAEMON section, which seems to contradict the message about mon.ceph-6 being stray. Any suggestions how to fix this? Thanks, Vlad

1 0

mgr+Prometheus/grafana (+consul)
by Jeremy Austin 20 May '21

20 May '21

I recently configured Prometheus to scrape mgr /metrics and add Grafana dashboards. All daemons at 15.2.11 I use Hashicorp consul to advertise the active mgr in DNS, and Prometheus points at a single DNS target. (Is anyone else using this method, or just statically pointing Prometheus at all potentially active managers?) All was working fine initially, and it's *mostly* still working fine. For the first couple of days, all went well, and then a few rate metrics stopped meaningfully increasing — essentially pegged at zero, which is implausible in a healthy cluster. Some cluster maintenance was occurring such as outing and recreating some OSDs, so I have a baseline for throughput and recovery. Metric graphs that stopped functioning: Throughput: ceph_osd_op_r_out_bytes, ceph_osd_op_w_in_bytes, ceph_osd_op_rw_in_bytes Recovery: ceph_osd_recovery_ops I can see that Grafana output is using this method of converting the counters to rates: sum(irate(ceph_osd_recovery_ops{job="$job"}[$interval])) The underlying counters appear to be sane, and reading the raw values from prometheus is also valid, so I'm guessing some failure either of the irate or sum functions? By inspection in Grafana, the queries return correct timestamps with zero values, so that leaves us with "sum(irate)" as the likely source of the problem. Does anyone have experience with this? I admit it is possibly tangential to ceph itself, but as the Prometheus/grafana integration is more or less supported, I thought I'd try here first. -- Jeremy Austin jhaustin(a)gmail.com

1 0

fsck error: found stray omap data on omap_head
by Pickett, Neale T 20 May '21

20 May '21

We just upgraded to pacific, and I'm trying to clear warnings about legacy bluestore omap usage stats by running 'ceph-bluestore-tool repair`, as instructed by the warning message. It's been going fine, but we are now getting this error: [root@vanilla bin]# ceph-bluestore-tool repair --path $osd_path 2021-05-19T19:25:26.485+0000 7f67ca3593c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) fsck error: found stray omap data on omap_head 12256434 0 0 repair status: remaining 1 error(s) and warning(s) [root@vanilla bin]# ceph-bluestore-tool fsck --path $osd_path -deep 2021-05-19T20:03:17.002+0000 7f4d1d6603c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) fsck error: found stray omap data on omap_head 12256434 0 0 fsck status: remaining 1 error(s) and warning(s) We're only 10% of the way through our OSDs, so I'd like to find some way to fix this other than destroying and rebuilding the OSD, in case it happens again. Fixing this error is especially attractive since we can't get out of HEALTH_WARN until we've run recover on all OSDs. Any suggestions? Neale Pickett <neale(a)lanl.gov> A-4: Advanced Research in Cyber Systems Los Alamos National Laboratory

2 2

"radosgw-admin bucket radoslist" loops when a multipart upload is happening
by Boris Behrens 20 May '21

20 May '21

Hi together, I still search for orphan objects and came across a strange bug: There is a huge multipart upload happening (around 4TB), and listing the rados objects in the bucket loops over the multipart upload. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 2

Bucket index OMAP keys unevenly distributed among shards
by James, GleSYS 20 May '21

20 May '21

Hi, we're running 15.2.7 and our cluster is warning us about LARGE_OMAP_OBJECTS (1 large omap objects). Here is what the distribution looks like for the bucket in question, and as you can see all but 3 of the keys reside in shard 2. .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.0 1 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.8 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.9 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.7 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.1 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.4 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.3 1 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.2 262384 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.6 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.5 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.12 0 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.10 1 .dir.5a5c812a-3d31-4d79-87e6-1a17206228ac.18635192.221.11 0 osd_deep_scrub_large_omap_object_key_threshold is set to 200000 by default, hence the warning observed for this bucket. Dynamic resharding is enabled, and the bucket is not in the process of being resharded. Versioning not in use for this bucket, so we're not affected by https://tracker.ceph.com/issues/46456. Can anyone help us understand why all the keys are getting mapped to a singe shard? Is there a bug here, or is this expected behaviour? Could it be related to the fact that the bucket contains large multipart uploads? (Object names look like this:) _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5900 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5901 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5902 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5903 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5904 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5905 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5906 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5907 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5908 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~0W-YhP3F7qc70Ad8JoBIugKzu225qs2.5909 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7152 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7153 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7154 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7155 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7156 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7157 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7158 _multipart_TOOTHROT/anonymised/TOOTHROT-DISK1-8c59002f-cffd-4f74-a680-147383ab8d78.vhdx.2~2uuwqny_HicO6kx_lPmWEf0zoyvdm_9.7159

1 0

ceph-ansible in Pacific and beyond?
by Matthew Vernon 20 May '21

20 May '21

Hi, I caught up with Sage's talk on what to expect in Pacific ( https://www.youtube.com/watch?v=PVtn53MbxTc ) and there was no mention of ceph-ansible at all. Is it going to continue to be supported? We use it (and uncontainerised packages) for all our clusters, so I'd be a bit alarmed if it was going to go away... Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

17 25

ceph orch status hangs forever
by Sebastian Luna Valero 20 May '21

20 May '21

Hi, After an unschedule power outage our Ceph (Octopus) cluster reports a healthy state with: "ceph status". However, when we run "ceph orch status" the command hangs forever. Are there other commands that we can run for a more thorough health check of the cluster? After looking at: https://docs.ceph.com/en/octopus/rados/operations/health-checks/ I also run "ceph crash ls-new" but it hangs forever as well. Any ideas? Our Ceph cluster is currently used as backend storage for our OpenStack cluster, and we are also having issues with storage volumes attached to VMs, but we don't know how to narrow down the root cause. Any feedback is highly appreciated. Best regards, Sebastian

2 3

Suitable 10G Switches for ceph storage - any recommendations?
by Hermann Himmelbauer 20 May '21

20 May '21

Dear Ceph users, I am currently constructing a small hyperconverged Proxmox cluster with ceph as storage. So far I always had 3 nodes, which I directly linked together via 2 bonded 10G network interfaces for the Ceph storage, so I never needed any switching devices. This new cluster has more nodes, so I am considering using a 10G switch for the storage network. As I have no experience with such a setup, I wonder if there are any specific issues that I should think of (latency...)? As the whole cluster should be not too expensive, I am currently thinking of the following solution: 2* CRS317-1G-16s+RM switches: https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults SFP+ Cables like these: https://www.fs.com/de/products/48883.html Some network interface for each node with two SFP+ ports, e.g.: https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-c… Connect each port with each switch and configure master/slave configuration so that the switches are redundant. What do you think of this setup - or is there any information / recommendation for an optimized setup of a 10G storage network? Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

2 1

iSCSI - failed, gateway(s) unavailable UNKNOWN
by Paul Giralt (pgiralt) 19 May '21

19 May '21

I’m new to Ceph and have just deployed my first cluster using ceph-ansible and running into some issues I was hoping someone could point me in the right direction. I have 5 servers to start with. 5 OSD, 3 Monitors, 2 Manager, 4 iSCSI gateway. My intention is to use this environment as iSCSI storage for ESXi. When I went to add an iSCSI Target, I got an error that popped up in the browser (went away to fast to read it) but after this, now if I click on the “Targets” tab in Dashboard for iSCSI, I just continuously get errors popping up: 500 - Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. I’m unable to add any targets now. The log files show this: 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 907, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 266, in list IscsiTarget._set_info(target) File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 990, in _set_info raise e File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 980, in _set_info target_iqn) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 531, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/iscsi_client.py", line 254, in get_targetinfo return request() File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 326, in __call__ data, raw_content, headers) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 449, in do_request resp.content) dashboard.rest_client.RequestException: iscsi REST API failed request with status code 503 (b'{\n "message": "failed, gateway(s) unavailable:cxcto-c240-j27-01(UNKNOWN' b' state)"\n}\n') 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR request] [10.117.244.166:58270] [GET] [500] [0.398s] [admin] [513.0B] /api/iscsi/target 2021-05-19T12:58:05.294-0400 7f2db6115700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "7bc691e8-e814-48c1-a69c-57ea1f3f3dbd"} ‘] Clearly it’s showing an error indicating that the gateway is “unavailable”. The output from gwcli shows this: [root@cxcto-c240-j27-01 ~]# ./gwcli 1 gateway is inaccessible - updates will be disabled /> ls o- / ......................................................................................................................... [...] o- cluster ......................................................................................................... [Clusters: 1] | o- ceph ............................................................................................................ [HEALTH_OK] | o- pools .......................................................................................................... [Pools: 4] | | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y] | | o- iscsi ............................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 262511b] | | o- rbd ................................................................... [(x3), Commit: 0.00Y/15911599M (0%), Used: 1650b] | | o- test .................................................................. [(x3), Commit: 0.00Y/15911599M (0%), Used: 0.00Y] | o- topology .............................................................................................. [OSDs: 110,MONs: 3] o- disks ....................................................................................................... [0.00Y, Disks: 0] o- iscsi-targets ............................................................................... [DiscoveryAuth: CHAP, Targets: 1] o- iqn.2001-07.com.ceph:1621437640904 ................................................................ [Auth: None, Gateways: 1] o- disks .......................................................................................................... [Disks: 0] o- gateways ............................................................................................ [Up: 0/1, Portals: 1] | o- cxcto-c240-j27-01 ....................................................................... [10.122.242.196 (UNAUTHORIZED)] o- host-groups .................................................................................................. [Groups : 0] o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0] In this case it says “UNAUTHORIZED” When I created the target, I had selected all 4 iSCSI gateways, but it looks like something happened during the addition process that has left it in a weird state. The dashboard seems to think that all the gateways are up and running : [cid:8CA421EB-DFD1-43A9-87B6-7ADEE5B2DF69] Notice that it only shows the target on the one node which it is reporting an error. I’ve tried deleting the target from gwcli but that fails and I’m not really sure where to look next. The “UNAUTHORIZED” in the gwcli output makes me wonder if there is some kind of authorization issue, but I’m not sure what that would be. This is what I see if I navigate to the Targets tab on the iSCSI page: [cid:828103A0-C321-41F6-B018-AC8A883F6C7E] Any thoughts or guidance are greatly appreciated. -Paul

1 0

Re: Suitable 10G Switches for ceph storage - any recommendations?
by Max Vernimmen 19 May '21

19 May '21

Hermann, I think there was a discussion on recommended switches not too long ago. You should be able to find it in the mailing list archives. I think the latency of the network is usually very minor compared to ceph's dependency on cpu and disk latency, so for a simple cluster I wouldn't worry about it too much. I have found fs.com's dac cables to get stuck a lot, so I don't use them anymore. I usually buy dell or mellanox cables. Regarding network cards I've found the intel cards to be not that great due to bugs with lacp bonds, embedded lldp getting in the way and other issues. So I'm using mellanox cards instead, but broadcom should also work. hope it helps! best regards, Max On Wed, May 19, 2021 at 1:48 PM <ceph-users-request(a)ceph.io> wrote: > ---------- Forwarded message ---------- > From: Hermann Himmelbauer <hermann(a)qwer.tk> > To: ceph-users(a)ceph.com > Cc: > Bcc: > Date: Wed, 19 May 2021 11:22:26 +0200 > Subject: [ceph-users] Suitable 10G Switches for ceph storage - any > recommendations? > Dear Ceph users, > I am currently constructing a small hyperconverged Proxmox cluster with > ceph as storage. So far I always had 3 nodes, which I directly linked > together via 2 bonded 10G network interfaces for the Ceph storage, so I > never needed any switching devices. > > This new cluster has more nodes, so I am considering using a 10G switch > for the storage network. As I have no experience with such a setup, I > wonder if there are any specific issues that I should think of > (latency...)? > > As the whole cluster should be not too expensive, I am currently > thinking of the following solution: > > 2* CRS317-1G-16s+RM switches: > https://mikrotik.com/product/crs317_1g_16s_rm#fndtn-testresults > > SFP+ Cables like these: > https://www.fs.com/de/products/48883.html > > Some network interface for each node with two SFP+ ports, e.g.: > > https://ark.intel.com/content/www/de/de/ark/products/39776/intel-ethernet-c… > > Connect each port with each switch and configure master/slave > configuration so that the switches are redundant. > > What do you think of this setup - or is there any information / > recommendation for an optimized setup of a 10G storage network? > > Best Regards, > Hermann > > -- > hermann(a)qwer.tk > PGP/GPG: 299893C7 (on keyservers) > >

2 1

rbd-nbd crashes Error: failed to read nbd request header: (33) Numerical argument out of domain
by Zhi Zhang 19 May '21

19 May '21

Hi guys, We are recently testing rbd-nbd using ceph N version. After map rbd image, mkfs and mount the nbd device, the rbd-nbd and dmesg will show following errors when doing some read/write testing. rbd-nbd log: 2021-05-18 11:35:08.034 7efdb8ff9700 20 []rbd-nbd: reader_entry: waiting for nbd request ... 2021-05-18 11:35:08.066 7efdb8ff9700 -1 []rbd-nbd: failed to read nbd request header: (33) Numerical argument out of domain 2021-05-18 11:35:08.066 7efdb3fff700 20 []rbd-nbd: writer_entry: no io requests, terminating 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b3c0 unregister_update_watcher: handle=0 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: handle=0 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b4b0 ImageUpdateWatchers::unregister_watcher: completing unregister 2021-05-18 11:35:08.066 7efdea8d1a00 10 []rbd-nbd: ~NBDServer: terminating 2021-05-18 11:35:08.066 7efdea8d1a00 20 []librbd::ImageState: 0x564a2be2b3c0 close dmesg: [Tue May 18 11:35:07 2021] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: discard [Tue May 18 11:35:07 2021] block nbd0: shutting down sockets [Tue May 18 11:35:09 2021] blk_update_request: I/O error, dev nbd0, sector 75592 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0 client host info: centos7.x kernel 5.4.109 It looks like the kernel nbd device shutdown its socket for some reason, but we haven't figured it out. BTW, we have tried to turn on/off rbd cache, use different fs ext4/xfs, use ec pool or replicated pool, but the error remains. It is more frequent for us to reproduce when batch map, mkfs and mount rbd-nbd on different hosts simultaneously. Thanks for any suggestions. Regards, Zhi Zhang (David) Contact: zhang.david2011(a)gmail.com zhangz.david(a)outlook.com

2 4

BlueFS spillover detected - 14.2.16
by Toby Darling 19 May '21

19 May '21

Hi In the last couple of weeks we've been getting BlueFS spillover warnings on multiple (>10) osds, eg BLUEFS_SPILLOVER BlueFS spillover detected on 1 OSD(s) osd.327 spilled over 58 MiB metadata from 'db' device (30 GiB used of 66 GiB) to slow device I know this can be corrected with a 'ceph tell osd.$osd compact' or ignored with "bluestore_warn_on_bluefs_spillover=false", but my concern is that these warnings have only recently started. Could this be a sign of something nasty heading our way that I'm not aware of? Is there a performance penalty by just ignoring, rather than compacting? Many thanks for any pointers. Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology

2 1

remove host from cluster for re-installing it
by mabi 19 May '21

19 May '21

Hello, On my Octopus cluster with 6 nodes (3 mon/mgr, 3 OSD), I would like to re-install the operating system of the first mon/mgr node. For that purpose I tried "ceph host rm mynode" but then I got the following two health warnings: 2 stray daemon(s) not managed by cephadm 1 stray host(s) with 2 daemon(s) not managed by cephadm So I did not proceed with the re-installation and added the node back. What would be the correct command in order to do that? I can live with mon/mgr which are do not have the quorum for the time of the re-installation. I just need my CephFS to still be available. Right now there is 2 mon daemons running and 1 active mgr and 2 standby mgrs. Thank you, Mabi

2 1

MDS process large memory consumption
by Andres Rojas Guerrero 19 May '21

19 May '21

Hi all, I have observed that in a Nautilus (14.2.6) cluster the mds process in the MDS server is consuming a large amount of memory, for example in a MDS server with 128 GB of RAM I have observed the mds process it's consuming ~ 80 GB: ceph 20 0 78,8g 77,1g 13772 S 4,0 61,5 28:37.62 ceph-mds Sometimes I have observed consumption of 127 GB, so the MDS starts to give slow request errors. In theory in the configuration I have limited the mds_cache_memory_limit to a 64 GB: [mds] mds_cache_memory_limit = 68719476736 client_oc_size = 104857600 mds_min_caps_per_client = 200 mds_log_max_expiring = 200 mds_log_max_segments = 250 What could be the reason for this excessive memory consumption? -- ******************************************************* Andrés Rojas Guerrero Unidad Sistemas Linux Area Arquitectura Tecnológica Secretaría General Adjunta de Informática Consejo Superior de Investigaciones Científicas (CSIC) Pinar 19 28006 - Madrid Tel: +34 915680059 -- Ext. 990059 email: a.rojas(a)csic.es ID comunicate.csic.es: @50852720l:matrix.csic.es *******************************************************

1 0

Pool has been deleted before snaptrim finished
by Szabo, Istvan (Agoda) 19 May '21

19 May '21

Hi, We decided to delete the pool before the snaptrim finished after 4 days waiting. Now we have bigger issue, many osd started to flap, 2 of them cannot even restart due after. Did some bluestore fsck on the not started osds and has many messages like this inside: 2021-05-17 18:37:07.176203 7f416d20bec0 10 stupidalloc 0x0x564e4e804f50 init_add_free 0x482d0778000~4000 2021-05-17 18:37:07.176204 7f416d20bec0 10 freelist enumerate_next 0x482d0784000~4000 2021-05-17 18:37:07.176204 7f416d20bec0 10 stupidalloc 0x0x564e4e804f50 init_add_free 0x482d0784000~4000 2021-05-17 18:37:07.176205 7f416d20bec0 10 freelist enumerate_next 0x482d078c000~c000 2021-05-17 18:37:07.176206 7f416d20bec0 10 stupidalloc 0x0x564e4e804f50 init_add_free 0x482d078c000~c000 [root@hk-cephosd-2002 ~]# tail -f /tmp/ceph-osd-44-fsck.log 2021-05-17 18:39:16.466967 7f416d20bec0 20 bluefs _read_random read buffered 0x2cd6e8f~ed6 of 1:0x372e0700000+4200000 2021-05-17 18:39:16.467154 7f416d20bec0 20 bluefs _read_random got 3798 2021-05-17 18:39:16.467179 7f416d20bec0 10 bluefs _read_random h 0x564e4e658500 0x24d6e35~ee2 from file(ino 216551 size 0x43a382d mtime 2021-05-17 13:21:19.839668 bdev 1 allocated 4400000 extents [1:0x35bc7c00000+4400000]) 2021-05-17 18:39:16.467186 7f416d20bec0 20 bluefs _read_random read buffered 0x24d6e35~ee2 of 1:0x35bc7c00000+4400000 2021-05-17 18:39:16.467409 7f416d20bec0 20 bluefs _read_random got 3810 and uh oh, missing shared_blob I've set back buffered_io to false back because when restart the osds always had to wait to fix degraded pgs. Many of the SSDs are smashing at the moment on 100% and don't really know what to do to stop the process and bring back the 2 ssds :/ Some paste: https://justpaste.it/9bj3a Some metric (each column is 1 server metric, total 3 servers): How it is smashing the ssds: https://i.ibb.co/x3xm0Rj/ssds.png IOWAIT Super high due to ssd utilization: https://i.ibb.co/683TR9y/iowait.png Capacity seems coming back: https://i.ibb.co/mz4Lq2r/space.png Thank you the help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

MDS rank 0 damaged after update to 14.2.20
by Eugen Block 19 May '21

19 May '21

Hi *, I tried a minor update (14.2.9 --> 14.2.20) on our ceph cluster today and got into a damaged CephFS. It's rather urgent since noone can really work right now, so any quick help is highly appreciated. As for the update process I followed the usual update procedure, when all MONs were finished I started to restart the OSDs, but suddenly our cephfs got unresponsive (and still is). I believe these lines are the critical ones: ---snap--- -12> 2021-05-18 09:53:01.488 7f7e9ed82700 5 mds.beacon.mds01 received beacon reply up:replay seq 906 rtt 0 -11> 2021-05-18 09:53:01.624 7f7e9f583700 10 monclient: get_auth_request con 0x5608a5171600 auth_method 0 -10> 2021-05-18 09:53:03.732 7f7e94d6e700 -1 mds.0.journaler.mdlog(ro) try_read_entry: decode error from _is_readable -9> 2021-05-18 09:53:03.732 7f7e94d6e700 0 mds.0.log _replay journaler got error -22, aborting -8> 2021-05-18 09:53:03.732 7f7e94d6e700 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument -7> 2021-05-18 09:53:03.732 7f7e94d6e700 5 mds.beacon.mds01 set_want_state: up:replay -> down:damaged -6> 2021-05-18 09:53:03.732 7f7e94d6e700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 -5> 2021-05-18 09:53:03.732 7f7e94d6e700 10 log_client will send 2021-05-18 09:53:03.735824 mds.mds01 (mds.0) 1 : cluster [ERR] Error loading MDS rank 0: (22) Invalid argument -4> 2021-05-18 09:53:03.732 7f7e94d6e700 10 monclient: _send_mon_message to mon.ceph01 at v2:XXX.XXX.XXX.XXX:3300/0 -3> 2021-05-18 09:53:03.732 7f7e94d6e700 5 mds.beacon.mds01 Sending beacon down:damaged seq 907 -2> 2021-05-18 09:53:03.732 7f7e94d6e700 10 monclient: _send_mon_message to mon.ceph01 at v2:XXX.XXX.XXX.XXX:3300/0 -1> 2021-05-18 09:53:03.908 7f7e9ed82700 5 mds.beacon.mds01 received beacon reply down:damaged seq 907 rtt 0.176001 0> 2021-05-18 09:53:03.908 7f7e94d6e700 1 mds.mds01 respawn! ---snap--- These logs are from the attempt to bring the mds rank back up with ceph mds repaired 0 I attached a longer excerpt of the log files if it helps. Before trying anything from the disaster recovery steps I'd like to ask for your input since one can damage it even more. The current status is below, please let me know if more information is required. Thanks! Eugen ceph01:~ # ceph -s cluster: id: 655cb05a-435a-41ba-83d9-8549f7c36167 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged noout flag(s) set Some pool(s) have the nodeep-scrub flag(s) set services: mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 116m) mgr: ceph03(active, since 118m), standbys: ceph02, ceph01 mds: cephfs:0/1 3 up:standby, 1 damaged osd: 32 osds: 32 up (since 64m), 32 in (since 8w) flags noout data: pools: 14 pools, 512 pgs objects: 5.08M objects, 8.6 TiB usage: 27 TiB used, 33 TiB / 59 TiB avail pgs: 512 active+clean

2 4

Re: Process for adding a separate block.db to an osd
by Boris Behrens 19 May '21

19 May '21

Hi, sorry for replying to this old thread: I tried to add a block.db to an OSD but now the OSD can not start with the error: Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]: -7> 2021-05-17 09:50:38.362 7fc48ec94a80 -1 rocksdb: Corruption: CURRENT file does not end with newline Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]: -6> 2021-05-17 09:50:38.362 7fc48ec94a80 -1 bluestore(/var/lib/ceph/osd/ceph-68) _open_db erroring opening db: Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]: -1> 2021-05-17 09:50:38.866 7fc48ec94a80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.21/rpm/el7/BUILD/ceph-14.2.21/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_upgrade_super()' thread 7fc48ec94a80 time 2021-05-17 09:50:38.865204 Mai 17 09:50:38 s3db10.fra2.gridscale.it ceph-osd[26038]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.21/rpm/el7/BUILD/ceph-14.2.21/src/os/bluestore/BlueStore.cc: 10647: FAILED ceph_assert(ondisk_format > 0) I tried to run an fsck/repair on the disk: [root@s3db10 osd]# ceph-bluestore-tool --path ceph-68 repair 2021-05-17 10:05:25.695 7f714dea3ec0 -1 rocksdb: Corruption: CURRENT file does not end with newline 2021-05-17 10:05:25.695 7f714dea3ec0 -1 bluestore(ceph-68) _open_db erroring opening db: error from fsck: (5) Input/output error [root@s3db10 osd]# ceph-bluestore-tool --path ceph-68 fsck 2021-05-17 10:05:35.012 7fb8f22e6ec0 -1 rocksdb: Corruption: CURRENT file does not end with newline 2021-05-17 10:05:35.012 7fb8f22e6ec0 -1 bluestore(ceph-68) _open_db erroring opening db: error from fsck: (5) Input/output error These are the steps I did to add the disk: $ CEPH_ARGS="--bluestore-block-db-size 53687091200 --bluestore_block_db_create=true" ceph-bluestore-tool bluefs-bdev-new-db --path /var/lib/ceph/osd/ceph-68 --dev-target /dev/sdj1 $ chown -h ceph:ceph /var/lib/ceph/osd/ceph-68/block.db $ lvchange --addtag ceph.db_device=/dev/sdj1 /dev/ceph-3bbfd168-2a54-4593-a037-80d0d7e97afd/osd-block-aaeaea54-eb6a-480c-b2fd-d938e336c0f6 $ lvchange --addtag ceph.db_uuid=463dd37c-fd49-4ccb-849f-c5827d3d9df2 /dev/ceph-3bbfd168-2a54-4593-a037-80d0d7e97afd/osd-block-aaeaea54-eb6a-480c-b2fd-d938e336c0f6 $ ceph-volume lvm activate --all The UUIDs later I tried this: $ ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-68 --devs-source /var/lib/ceph/osd/ceph-68/block --dev-target /var/lib/ceph/osd/ceph-68/block.db bluefs-bdev-migrate Any ideas how I can get the rocksdb fixed? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 13

Ceph increase RBD Pool Size not change
by codignotto 18 May '21

18 May '21

I am increasing the size of my POOL from 24TiB to 28TiB, I execute the change via the ceph portal, it shows that everything is ok but it does not change the real value, it remains in the same 24TiB, would it be a Bug? I was managing to increase up to 24TiB, I can increase it to 34Tb q is the size of the cluster with replica 2 but it is not running . Any idea ? Tks

1 0

image + snapshot remove
by Szabo, Istvan (Agoda) 18 May '21

18 May '21

Hi, Pool has been deleted in which was an image that related to another pools and now can't remove the image due to it has snapshot. Can't list snapshot, can't purge, can't flattened can't really do anything. Is it possible to remove the image without pool delete? 021-05-18 14:55:26.710 7f8bef381c80 -1 librbd::image::PreRemoveRequest: 0x56301203ac00 check_image_snaps: image has snapshots - not removing Removing image: 0% complete...failed. 2021-05-18 14:55:26.722 7f8bccff9700 -1 librbd::image::RefreshRequest: failed to refresh parent image: (2) No such file or directory 2021-05-18 14:55:26.723 7f8bc7fff700 -1 librbd::image::OpenRequest: failed to refresh image: (2) No such file or directory rbd: error opening image clone_prod_financedb_lv1_20210513_160225: (2) No such file or directory rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. Or we need to find the original image and upload back? Thx ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

logrotation in ceph 16.2.4
by Fabrice Bacchella 18 May '21

18 May '21

I have a ceph cluster with 6 OSD servers. 2 are running 16.2.4 and logrotate failed with this message: etc/cron.daily/logrotate: error: Compressing program wrote following message to stderr when compressing log /var/log/ceph/ceph-osd.37.log-20210518: gzip: stdin: file size changed while zipping The four 4 16.2.3 OSD servers went fine. The /etc/logrotate.d/ceph file is the same between versions. I think there might be a bug with the handling of signal SIGHUP (-1)

1 0

libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
by Markus Kienast 17 May '21

17 May '21

I am seeing these messages when booting from RBD and booting hangs there. libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks. How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented. This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem. Cheers Markus

4 14

After a huge amount of snaphot delete many snaptrim+snaptrim_wait pgs
by Szabo, Istvan (Agoda) 17 May '21

17 May '21

Hi, The user deleted 20-30 snapshots and clones from the cluster and it seems like slows down the whole system. I’ve set the snaptrim parameters to the lowest as possible, set bufferred_io to true so at least have some speed for the user, but I can see the objects removal from the cluster is still happening, the beginning was 45 millioms, now 19millions but what I don’t understand many osds getting more full :( ? And the snaptrim is super slow, have 195 snaptrim wait and 36 snaptrim but in every 5 hours only 1 done :/? What can I do? One of the osd has been 62%, now it is 75% in 2 days and still growing. Set back the snap options or? The cluster has 3 servers, running on luminous 12.2.8. Some paste: https://jpst.it/2vw4H Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

RGW: Multiple Site does not sync olds data
by 特木勒 17 May '21

17 May '21

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

3 17

bluefs_buffered_io turn to true
by Szabo, Istvan (Agoda) 17 May '21

17 May '21

Hi, I had issue with the snaptrim after a hug amount of deleted data, it slows down the team operations due to the snaptrim and snaptrim_wait pgs. I've changed couple of things: debug_ms = 0/0 #default 0/5 osd_snap_trim_priority = 1 # default 5 osd_pg_max_concurrent_snap_trims = 1 # default 2 But didn't help. I've found this thread about buffered io and seems like it helped to them: https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-night… I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused basically the complete node's pg-s are degraded. Is it normal? I hope it will not rebalance the complete node because I don't have space for that. I changed it back but still slowly decreasing, so not sure this settings is correct or not or this behavior is good or not? 2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 objects misplaced (0.004%) (OBJECT_MISPLACED) 2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data redundancy: 33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 pgs undersized (PG_DEGRADED) Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 17

Octopus MDS hang under heavy setfattr load
by Nigel Williams 17 May '21

17 May '21

One of my colleagues attempted to set quotas on a large number (some dozens) of users with the session below, but it caused the MDS to hang and reject client requests. Offending command was: cat recent-users | xargs -P16 -I% setfattr -n ceph.quota.max_bytes -v 8796093022208 /scratch/% Result was to hang /scratch and any other mounts managed by the same MDS on all clients. Status of ceph-mds while broken was: root@cnx-14:~# systemctl status ceph-mds@cnx-14 ● ceph-mds(a)cnx-14.service - Ceph metadata server daemon Loaded: loaded (/lib/systemd/system/ceph-mds@.service; indirect; vendor preset: enabled) Active: active (running) since Thu 2021-05-06 17:16:45 AEST; 1 weeks 3 days ago Main PID: 2385 (ceph-mds) Tasks: 23 CGroup: /system.slice/system-ceph\x2dmds.slice/ceph-mds(a)cnx-14.service └─2385 /usr/bin/ceph-mds -f --cluster ceph --id cnx-14 --setuser ceph --setgroup cephMay 13 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-13T06:25:01.724+1000 7f5444832700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-m May 13 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-13T06:25:01.736+1000 7f5444832700 -1 received signal: Hangup from (PID: 229281) UID: 0 May 14 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-14T06:25:01.992+1000 7f5444832700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-m May 14 06:25:02 cnx-14 ceph-mds[2385]: 2021-05-14T06:25:02.004+1000 7f5444832700 -1 received signal: Hangup from (PID: 232464) UID: 0 May 15 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-15T06:25:01.468+1000 7f5444832700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-m May 15 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-15T06:25:01.480+1000 7f5444832700 -1 received signal: Hangup from (PID: 236005) UID: 0 May 16 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-16T06:25:01.989+1000 7f5444832700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-m May 16 06:25:02 cnx-14 ceph-mds[2385]: 2021-05-16T06:25:02.001+1000 7f5444832700 -1 received signal: Hangup from (PID: 239260) UID: 0 May 17 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-17T06:25:01.813+1000 7f5444832700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-m May 17 06:25:01 cnx-14 ceph-mds[2385]: 2021-05-17T06:25:01.829+1000 7f5444832700 -1 received signal: Hangup from (PID: 242044) UID: 0 Fix was to run: systemctl restart ceph-mds@cnx-14 Non parallelised run of xargs with sleep 1 between each iteration worked.

1 0

Re: Limit memory of ceph-mgr
by mabi 17 May '21

17 May '21

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, May 17, 2021 4:31 AM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > You’re running on so small a node that 3.6GB is a problem?? Yes, I have hardware constraints based on the hardware where my hardware has maximum 8 GB of RAM as it is a Raspberry Pi 4. I am doing a proof of concept to see if it is possible to run a small cluster on this type of hardware. So it is no problem to me to scale horizontally (add more nodes) but scaling vertically (adding more RAM or CPU) is not possible. It is also a great way of learning and experiencing ceph and see what can be done in terms of optimization.

1 0

after upgrade to 16.2.3 16.2.4 and after adding few hdd's OSD's started to fail 1 by 1.
by Andrius Jurkus 17 May '21

17 May '21

Hello, I will try to keep it sad and short :) :( PS sorry if this dublicate I tried post it from web also. Today I upgraded from 16.2.3 to 16.2.4 and added few hosts and osds. After data migration for few hours, 1 SSD failed, then another and another 1 by 1. Now I have cluster in pause and 5 failed SSD's, same host has SSD and HDD, but only SSD's are failing so I think this has to be balancing refiling or something bug and probably not upgrade bug. Cluster has been in pause for 4 hours and no more OSD's are failing. full trace https://pastebin.com/UxbfFYpb Now I m googling and learning but, Is there a way how to easily test lets say 15.2.XX version on osd without losing anything? Any help would be appreciated. Error start like this May 14 16:58:52 dragon-ball-radar systemd[1]: Starting Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11... May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.057836433 +0000 UTC m=+0.454352919 container create 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, maintainer=D May 14 16:58:53 dragon-ball-radar systemd[1]: Started libcrun container. May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.3394116 +0000 UTC m=+0.735928098 container init 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, maintainer=Dimitri Savineau <dsav May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.446921192 +0000 UTC m=+0.843437626 container start 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, GIT_BRANCH=HEAD, org.label-sch May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.447050119 +0000 UTC m=+0.843566553 container attach 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, org.label-schema.name=CentOS May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 --path /var/lib/ceph/osd/ceph-2 --no-mon-config May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/ln -snf /dev/ceph-45e6ef2e-fbdc-4289-a900-3d1ffc81ee14/osd-block-973cfe73-06c8-4ea0-9aea-1361d063eb25 /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 May 14 16:58:53 dragon-ball-radar bash[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 May 14 16:58:53 dragon-ball-radar bash[113558]: --> ceph-volume lvm activate successful for osd ID: 2 May 14 16:58:53 dragon-ball-radar podman[113650]: 2021-05-14 16:58:53.8147653 +0000 UTC m=+1.211281741 container died 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate) May 14 16:58:55 dragon-ball-radar podman[113650]: 2021-05-14 16:58:55.044964534 +0000 UTC m=+2.441480996 container remove 3b44520aa651b8196cd0bf0c96daa2bd03845ef5f8cfaf9a689410a1f98d84dd (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2-activate, CEPH_POINT_RELEASE=-16.2.4, R May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.594265612 +0000 UTC m=+0.369978347 container create 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, RELEASE=HEAD, org.label-schema.build-d May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.864589286 +0000 UTC m=+0.640302021 container init 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, org.label-schema.schema-version=1.0, GIT May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 set uid:gid to 167:167 (ceph:ceph) May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-osd, pid 2 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 0 pidfile_write: ignore empty --pid-file May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.896+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block size 466 GiB May 14 16:58:55 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:55.900+0000 7fcf16aa2080 1 bdev(0x564ad3a8cc00 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:55 dragon-ball-radar podman[113909]: 2021-05-14 16:58:55.972267166 +0000 UTC m=+0.747979911 container start 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da (image=docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949, name=ceph-4e01640b-951b-4f75-8dca-0bad4faf1b11-osd.2, ceph=True, GIT_REPO=https://github.com/ May 14 16:58:55 dragon-ball-radar bash[113558]: 31364008fcb8b290643d6e892fba16d19618f5682f590373feabed23061749da May 14 16:58:55 dragon-ball-radar systemd[1]: Started Ceph osd.2 for 4e01640b-951b-4f75-8dca-0bad4faf1b11. May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.184+0000 7fcf16aa2080 1 bdev(0x564ad3a8c800 /var/lib/ceph/osd/ceph-2/block) close May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 1 objectstore numa_node 0 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 0 starting osd.2 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '10.0.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.444+0000 7fcf16aa2080 -1 unable to find any IPv4 address in networks '172.16.199.0/24' interfaces '' May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.452+0000 7fcf16aa2080 0 load: jerasure load: lrc load: isa May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) open size 500103643136 (0x7470800000, 466 GiB) block_size 4096 (4 KiB) non-rotational discard supported May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 May 14 16:58:56 dragon-ball-radar conmon[113957]: debug 2021-05-14T16:58:56.456+0000 7fcf16aa2080 1 bdev(0x564ad476e400 /var/lib/ceph/osd/ceph-2/block) close

4 4

v16.2.4 Pacific released
by David Galloway 16 May '21

16 May '21

This is a hotfix release addressing a number of security issues and regressions. We recommend all users update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v16-2-4-pacific-released Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.4.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 3cbe25cde3cfa028984618ad32de9edc4c1eaed0

2 1

CephFS Snaptrim stuck?
by Andras Sali 16 May '21

16 May '21

Dear Ceph Users, We are experiencing a strange behaviour on Ceph v15.2.9 that a set of PGs seem to be stuck in active + clean + snaptrim state. (for almost a day now) Usually snaptrim is quite fast (done in a few minutes), however now in the osd logs we see slowly increasing trimq numbers, with such entries coming constantly (every few seconds): 2021-05-16T13:58:28.795+0000 7fc668f2e700 -1 osd.9 pg_epoch: 91137 pg[2.39( v 91137'1584600 (91059'1581379,91137'1584600] local-lis/les=91119/91120 n=115714 ec=165/115 lis/c=91119/91119 les/c/f=91120/91120/0 sis=91119) [9,10,3] r=0 lpr=91119 luod=91137'1584598 crt=91137'1584600 lcod 91137'1584597 mlcod 91137'1584597 active+clean+snaptrim* trimq=82* ps=[75fe~1,7600~1,868a~1,8caa~1,a30c~1,a422~1,a65e~1,c0cf~1,c569~1]] removing snap head We tried restarting the OSD-s, but no difference. Otherwise the cluster reports itself as being healthy. If anyone has any ideas what might be causing this and how to get these PGs finished with the snaptrim state, that would be very much appreciated. Kind regards, András Sali

1 0

dedicated metadata servers
by mabi 16 May '21

16 May '21

Hello, On my small Octopus 6 nodes cluster I have only 8 GB per node available and I am co-locating the active and standby MDS on two out of the 3 OSD nodes. But because memory is tight (8GB per node max, limited due to hardware constraints) I was thinking to add two new 8GB nodes dedicated only for MDS so they can benefit of more RAM. So my question is, is this a good idea? Are there any downsides of having MDS active and passive on two dedicated nodes? Regards, Mabi

2 1

radosgw lost config during upgrade 14.2.16 -> 21
by Jan Kasprzak 15 May '21

15 May '21

Hello, I have just upgraded my cluster from 14.2.16 to 14.2.21, and after the upgrade, radosgw was listening on the default port 7480 instead of the SSL port it used before the upgrade. It might be I mishandled "ceph config assimilate-conf" previously or forgot to restart radosgw after the assimilate-conf or something. What is the correct way to store radosgw configuration in ceph config? I have the following (which I think worked previously, but I might be wrong, e.g. forgot to restart radosgw or something): # ceph config dump [...] client.rgw.<myhost> basic rgw_frontends beast ssl_port=<myport> ssl_certificate=/etc/pki/tls/certs/<myhost>.crt+bundle ssl_private_key=/etc/pki/tls/private/<myhost>.key * However, after rgw startup, there was the following in /var/log/ceph/ceph-client.rgw.<myhost>.log: 2021-05-14 21:38:35.075 7f6ffd621900 1 mgrc service_daemon_register rgw.<myhost> metadata {arch=x86_64,ceph_release=nautilus,ceph_version=ceph version 14.2.21 (5ef401921d7a88aea18ec7558f7f9374ebd8f5a6) nautilus (stable),ceph_version_short=14.2.21,cpu=AMD ...,distro=centos,distro_description=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=beast port=7480,frontend_type#0=beast,hostname=<myhost>,kernel_description=#1 SMP ...,kernel_version=...,mem_swap_kb=...,mem_total_kb=...,num_handles=1,os=Linux,pid=20451,zone_id=...,zone_name=default,zonegroup_id=...,zonegroup_name=default} (note the port=7480 and no SSL). After adding the following into /etc/ceph/ceph.conf on the host where rgw is running, it started to use the correct SSL port again: [client.rgw.<myhost>] rgw_frontends = beast ssl_port=<myport> ssl_certificate=/etc/pki/tls/certs/<myhost>.crt+bundle ssl_private_key=/etc/pki/tls/private/<myhost>.key How can I configure this using "ceph config"? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 1