March 2021 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

does ceph rgw has any option to limit bandwidth
by Zhenshi Zhou 14 Jun '21

14 Jun '21

Hi, Is there any option of rados gateway that limit bandwidth?

5 6

Recommendations on problem with PG
by Gabriel Medve 23 May '21

23 May '21

Hi, We have a problem with a PG that was inconsistent, currently the PG in our cluster have 3 copies. It was not possible for us to repair this pg with "ceph pg repair" (This PG is in osd 14,1,2) so we deleted some of the copies of osd 14 with the following command. ceph-objectstore-tool --data-path /var/lib/ceph/osd.14/ --pgid 22.f --op remove --force This caused an automatic attempt to create the missing copy entering the backfilling state, but when doing this it crashed osd 1 and 2 and threw the IOPS to 0, freezing the cluster. Is there any way to remove this entire pg or try to recreate the missing copy or ignore it completely? It causes instability in the cluster. Thank you, I await comments -- Untitled Document ------------------------------------------------------------------------ Gabriel I. Medve

2 2

ceph-ansible in Pacific and beyond?
by Matthew Vernon 20 May '21

20 May '21

Hi, I caught up with Sage's talk on what to expect in Pacific ( https://www.youtube.com/watch?v=PVtn53MbxTc ) and there was no mention of ceph-ansible at all. Is it going to continue to be supported? We use it (and uncontainerised packages) for all our clusters, so I'd be a bit alarmed if it was going to go away... Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

17 25

RGW: Multiple Site does not sync olds data
by 特木勒 17 May '21

17 May '21

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to enable full sync for bucket or this is a bug? Thanks

3 17

Zabbix module Octopus 15.2.3
by Gert Wieberdink 14 May '21

14 May '21

Trying to configure Zabbix module in Octopus 15.2.3. CentOS 8.1 environment. Installed zabbix40-agent for CentOS 8.1 (from epel repository). This will also install zabbix_sender. After enabling the Zabbix module in Ceph, I configured my Zabbix host and Zabbix identifier. # ceph zabbix config-set zabbix_host <zabbix-fqdn> # ceph zabbix config-set zabbix_identifier <ident> # ceph zabbix config-show Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1153, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/zabbix/module.py", line 407, in handle_command return 0, json.dumps(self.config, index=4, sort_keys=True), '' File "/lib64/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) TypeError: __init__() got an unexpected keyword argument 'index' # ceph -v ceph version 15.2.3 (d289bbdec69ed7c1f516e0a093594580a76b78d0) octopus (stable) # ceph health detail HEALTH_OK Anyone found a solution? rgds, -gw

5 4

OSD id 241 != my id 248: conversion from "ceph-disk" to "ceph-volume simple" destroys OSDs
by Frank Schilder 04 May '21

04 May '21

Hi all, this is a follow-up on "reboot breaks OSDs converted from ceph-disk to ceph-volume simple". I converted a number of ceph-disk OSDs to ceph-volume using "simple scan" and "simple activate". Somewhere along the way, the OSDs meta-data gets rigged and the prominent symptom is that the symlink block is changes from a part-uuid target to an unstable device name target like: before conversion: block -> /dev/disk/by-partuuid/9123be91-7620-495a-a9b7-cc85b1de24b7 after conversion: block -> /dev/sdj2 This is a huge problem as the "after conversion" device names are unstable. I have now a cluster that I cannot reboot servers on due to this problem. OSDs randomly re-assigned devices will refuse to start with: 2021-03-02 15:56:21.709 7fb7c2549b80 -1 OSD id 241 != my id 248 Please help me with getting out of this mess. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 7

OSDs RocksDB corrupted when upgrading nautilus->octopus: unknown WriteBatch tag
by Jonas Jelten 27 Apr '21

27 Apr '21

Hi! After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10 shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot. RocksDB complains "Corruption: unknown WriteBatch tag". The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs". Tracker issue with logs: https://tracker.ceph.com/issues/50017 Anyone else encountered this error? I've "suspended" the upgrade for now :) -- Jonas

5 12

Ceph osd Reweight command in octopus
by Brent Kennedy 12 Apr '21

12 Apr '21

We have a ceph octopus cluster running 15.2.6, its indicating a near full osd which I can see is not weighted equally with the rest of the osds. I tried to do the usual "ceph osd reweight osd.0 0.95" to force it down a little bit, but unlike the nautilus clusters, I see no data movement when issuing the command. If I run a ceph osd tree, it shows the reweight setting, but no data movement appears to be occurring. Is there some new thing in ocotopus I am missing? I looked through the release notes for .7, .8 and .9 and didn't see any fixes that jumped out as resolving a bug related to this. The Octopus cluster was deployed using ceph-ansible and upgraded to 15.2.6. I plan to upgrade to 15.2.9 in the coming month. Any thoughts? Regards, -Brent Existing Clusters: Test: Ocotpus 15.2.5 ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(SSD): Octopus 15.2.6 with 5 osd servers, 3 mons, 4 gateways

2 2

upgrade problem nautilus 14.2.15 -> 14.2.18? (Broken ceph!)
by Simon Oosthoek 09 Apr '21

09 Apr '21

Hi I'm in a bit of a panic :-( Recently we started attempting to configure a radosgw to our ceph cluster, which was until now only doing cephfs (and rbd wss working as well). We were messing about with ceph-ansible, as this was how we originally installed the cluster. Anyway, it installed nautilus 14.2.18 on the radosgw and I though it would be good to pull up the rest of the cluster to that level as well using our tried and tested ceph upgrade script (it basically does an update of all ceph nodes one by one and checks whether ceph is ok again before doing the next) After the 3rd mon/mgr was done, all pg's were unavailable :-( obviously, the script is not continuing, but ceph is also broken now... The message deceptively is: HEALTH_WARN Reduced data availability: 5568 pgs inactive That's all PGs! I tried as a desperate measure to upgrade one ceph OSD node, but that broke as well, the osd service on that node gets an interrupt from the kernel.... the versions are now like: 20:29 [root@cephmon1 ~]# ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156 }, "mds": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158, "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6 } } 12 OSDs are down # ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN Reduced data availability: 5568 pgs inactive services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m) mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722 remapped pgs data: pools: 12 pools, 5568 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 5568 unknown progress: Rebalancing after osd.103 marked in [..............................]

5 11

Ceph CFP Coordination for 2021
by Mike Perez 08 Apr '21

08 Apr '21

Hi everyone, I cleaned up the CFP coordination etherpad with some events coming up. Please add other events you think the community should be considering proposing content on Ceph or adjacent projects like Rook. KubeCon NA CFP, for example, is ending April 11. Take a look: https://pad.ceph.com/p/cfp-coordination I have also added this to our wiki for discovery. https://tracker.ceph.com/projects/ceph/wiki/Community -- Mike Perez

1 1

cephadm and ha service for rgw
by Seba chanel 08 Apr '21

08 Apr '21

Hi everyone, I try to configure HA service for rgw with cephadm. I have 2 rgw on cnrgw1 et cnrgw2 for the same pool. i use a virtual IP address 192.168.0.15 cnrgwha and the config from https://docs.ceph.com/en/latest/cephadm/rgw/#high-availability-service-for-… # from root@cnrgw1 [root@cnrgw1 ~]# cat /etc/sysctl.conf net.ipv6.conf.all.disable_ipv6 = 1 net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1 [root@cnrgw1 ~]# sysctl -p net.ipv6.conf.all.disable_ipv6 = 1 net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1 #same from cnrgw2 #generate cert [vagrant@cn1 ~]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./rgwha.key -out ./rgwha.crt Generating a RSA private key .............+++++ ........................................................+++++ writing new private key to './rgwha.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:fr State or Province Name (full name) []:est Locality Name (eg, city) [Default City]:sbg Organization Name (eg, company) [Default Company Ltd]:cephlab.org Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []:cnrgwha Email Address []:root@localhost # write the YAML rgwha.yaml service_type: ha-rgw service_id: haproxy_for_rgw placement: hosts: - cnrgw1 - cnrgw2 spec: virtual_ip_interface: eth1 virtual_ip_address: 192.168.0.15/24 frontend_port: 8080 ha_proxy_port: 1967 ha_proxy_stats_enabled: true ha_proxy_stats_user: admin ha_proxy_stats_password: true ha_proxy_enable_prometheus_exporter: true ha_proxy_monitor_uri: /haproxy_health keepalived_user: admin keepalived_password: admin ha_proxy_frontend_ssl_certificate: [ "-----BEGIN CERTIFICATE-----", "MIICSzCCAfWgAwIBAgIUWKC9e+5tnIAjddECXOGc144p8E0wDQYJKoZIhvcNAQEL", "BQAwejELMAkGA1UEBhMCZnIxDDAKBgNVBAgMA2VzdDEMMAoGA1UEBwwDc2JnMRAw", "DgYDVQQKDAdjZXBobGFiMQwwCgYDVQQLDANvcmcxEDAOBgNVBAMMB2Nucmd3aGEx", "HTAbBgkqhkiG9w0BCQEWDnJvb3RAbG9jYWxob3N0MB4XDTIxMDMwOTE0MjI0N1oX", "DTIyMDMwOTE0MjI0N1owejELMAkGA1UEBhMCZnIxDDAKBgNVBAgMA2VzdDEMMAoG", "A1UEBwwDc2JnMRAwDgYDVQQKDAdjZXBobGFiMQwwCgYDVQQLDANvcmcxEDAOBgNV", "BAMMB2Nucmd3aGExHTAbBgkqhkiG9w0BCQEWDnJvb3RAbG9jYWxob3N0MFwwDQYJ", "KoZIhvcNAQEBBQADSwAwSAJBAMqji/AKBr6DbuHKOTWyIBWbeYkyZ7Jn7fqfZceE", "p7G321t1TvAjD7sa64FRT6n4x8CtzKPGXXpRr28o8oR1h70CAwEAAaNTMFEwHQYD", "VR0OBBYEFIQim5ZxojFny+srzQJIs1N8wLmYMB8GA1UdIwQYMBaAFIQim5ZxojFn", "y+srzQJIs1N8wLmYMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADQQCE", "eCwMQFNYtw+4I1QzTV13ewawuPkPdrhiNzcs0mgt93+quE0zBIeOY2jnFmlo6H/h", "syYGvwgcAh9VW9qo5fsk", "-----END CERTIFICATE-----", "-----BEGIN PRIVATE KEY-----", "MIIBVQIBADANBgkqhkiG9w0BAQEFAASCAT8wggE7AgEAAkEAyqOL8AoGvoNu4co5", "NbIgFZt5iTJnsmft+p9lx4SnsbfbW3VO8CMPuxrrgVFPqfjHwK3Mo8ZdelGvbyjy", "hHWHvQIDAQABAkB0kt2AO+RhWS9CyZlb4JtAku66FLs/ETcAxQ5CV3g5beq8/wRs", "x3xZhIsjdr7OZZ+BEoJYn+0upywoctXmwM8BAiEA+KG26RADqJfAdoRn640UrT9E", "pfF3drDrQg0WrKAf3N0CIQDQpOZa0pV2GL28u2NaU85uJCDeKDWhTnvFEqlLu/S4", "YQIhAPY+0/WIUtdLVOcMxA/bLrtXihoASR1Yo+hLJkXaYTRRAiB3Rh1txD6vEXu+", "Hb2xUIGNE1g6x+/ItA4rXfysD9nZYQIhAKYn3IdG55JwiwSKv8gVAEdX8xiUfEjY", "pnvk3p52VHHI", "-----END PRIVATE KEY-----" ] ha_proxy_frontend_ssl_port: 8090 ha_proxy_ssl_dh_param: 1024 ha_proxy_ssl_ciphers: ECDH+AESGCM:!MD5 ha_proxy_ssl_options: no-sslv3 haproxy_container_image: haproxy:2.4-dev3-alpine keepalived_container_image: arcts/keepalived:1.2.2 # apply the new config [ceph: root@cn1 ~]# ceph orch apply -i rgwha.yaml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'virtual_ip_interface' Do you have any leads why it doesn't work? [ceph: root@cn1 /]# ceph versions { "mon": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 5 }, "mgr": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 2 }, "osd": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 8 }, "mds": {}, "rgw": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 2 }, "overall": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 17 } }

2 2

understanding orchestration and cephadm
by Gary Molenkamp 05 Apr '21

05 Apr '21

Good morning all, I'm experimenting with ceph orchestration and cephadm after using ceph-deploy for several years, and I have a hopefully simple question. I've converted a basic nautilus cluster over to cephadm+orchestration and I tried adding, then removing a monitor. However, when I removed the host using 'ceph orch host rm', it removed two mons. I may have missed something in the adoption/upgrade that has left the cluster in a bad state. Any advice/pointers/clarification would be of assistance. Details: A nautilus cluster with two mons (I know this is not correct for quorum), a mgr, and a handful of osds. I went though the adoption process and enabled the ceph orch backend. [root@osdev-ctrl2 ~]# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f e73c19b51a09 mon.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f a6bfc27221f0 mon.osdev-net1 osdev-net1 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f f66e2bef3d44 osd.0 osdev-stor1 running (17h) 50s ago 17h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f ac59dbdc267c ... [root@osdev-ctrl2 ~]# ceph orch status Backend: cephadm Available: True [root@osdev-ctrl2 ~]# ceph orch host ls HOST ADDR LABELS STATUS osdev-ctrl2 osdev-ctrl2 mon mgr osdev-net1 osdev-net1 mon osdev-stor1 osdev-stor1 osd [root@osdev-ctrl2 ~]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID mgr 1/1 9m ago 20h label:mgr docker.io/ceph/ceph:v15.2.10 5b724076c58f mon 2/2 9m ago 20h label:mon docker.io/ceph/ceph:v15.2.10 5b724076c58f I then added a new mon host: [root@osdev-ctrl2 ~]# ceph orch host add osdev-ctrl3 mon It did not spawn a mon container on osdev-ctrl3 until I defined the public network in the config: [root@osdev-ctrl2 ~]# ceph config set global public_network 10.10.10.0/24 At this point all is good with three running mon as expected. Now I wanted to delete the mon using [root@osdev-ctrl2 ~]# ceph orch host rm osdev-ctrl3 This had the effect of: 1. removing the osdev-ctrl3 mon from 'ceph orch ls' and 'ceph orch ps' 2. the mon on osdev-ctrl3 is still running, and is part of 'ceph -s' but reported as not managed by cephadm 3. (Big issue) the mon running on osdev-net1 was completely destroyed. Any ideas what is going on? Sorry for the long post, but I tried to be as clear as possible. -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566

3 3

First 6 nodes cluster with Octopus
by mabi 05 Apr '21

05 Apr '21

Hello, I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and have a few questions mostly regarding planning of the infra. 1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to use Ubuntu 20.04 instead or should I stick with 18.04? 2) The documentation recommends using Cephadm for new deployments, so I will use that but I read that with Cephadm everything is running in containers, so is this the new way to go? Or is Ceph in containers kind of still experimental? 3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I am planning the following layout: Node 1: MGR+MON+MDS Node 2: MGR+MON+MDS Node 3: MGR+MON+MDS Node 4: OSD Node 5: OSD Node 6: OSD Does this make sense? I am mostly interested in stability and HA with this setup. 4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I understand the MDS will create a metadata pool on the OSDs. Thanks for the hints. Best, Mabi

4 7

Ceph User Survey Working Group - Next Steps
by Mike Perez 01 Apr '21

01 Apr '21

Hi everyone, We are approaching the April 2nd deadline in two weeks, so we should start proposing the next meeting to plan the survey results. Anybody in the community is welcome to join the Ceph Working Groups. Please add your name to: https://ceph.io/user-survey/ I have started a doodle: https://doodle.com/poll/y3t2ttdt8a3egz4v?utm_source=poll&utm_medium=link Please help promote the User Survey: https://twitter.com/Ceph/status/1369589099716349956 -- Mike Perez (thingee)

2 2

v14.2.19 Nautilus released
by David Galloway 01 Apr '21

01 Apr '21

This is the 19th update to the Ceph Nautilus release series. This is a hotfix release to prevent daemons from binding to loopback network interfaces. All nautilus users are advised to upgrade to this release. Notable Changes --------------- * This release fixes a regression introduced in v14.2.18 whereby in certain environments, OSDs will bind to 127.0.0.1. See https://tracker.ceph.com/issues/49938. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.19.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: bb796b9b5bab9463106022eef406373182465d11

3 5

Running ceph on multiple networks
by Andrei Mikhailovsky 31 Mar '21

31 Mar '21

Hello everyone, I have a small ceph cluster consisting of 4 Ubuntu 20.04 osd servers mainly serving rbd images to Cloudstack kvm cluster. The ceph version is 15.2.9. The network is done in such a way that all storage cluster is ran over infiniband qdr links (ipoib). We've got the management network for our ceph servers and kvm over ethernet (192.168.1.1/24) and the ipoib storage network 192.168.2.1/24. We are in the process of updating our cluster with new hardware and planning to scrap the infiniband connectivity altogether and replace it with 10gbit ethernet. We are also going to replace the kvm host servers too. We were hoping to have minimal or preferably no downtime in this process. I was wondering if we could run the ceph services (mon, osd, radosgw) concurrently over two networks after we've added the 10G ethernet? While the upgrades and migration taking place, we need to have the ceph running over the current ipoib 192.168.2.1/24 as well as the 10G 192.168.3.1/24. Could you please help me with this? Cheers Andrei

1 0

How's the maturity of CephFS and how's the maturity of Ceph erasure code?
by fanyuanli 31 Mar '21

31 Mar '21

Hi all, I'm a rookie in CEPH. I want to ask two questions. One is the maturity of cephfs, the file system of CEPH, and whether it is recommended for production environment. The other is the maturity of CEPH's erasure code and whether it can be used in production environment. Are the above two questions explained in the official documents? I may not see where they are. Thank you! Fred Fan

4 4

15.2.10 Dashboard incompatible with Reverse Proxy?
by Christoph Brüning 31 Mar '21

31 Mar '21

Hello everyone, we have updated one of our clusters from 15.2.9 to 15.2.10 and cannot access the dashboard any more. The dashboard is behind an nginx reverse proxy, which proxy_passes https://some.admin.hostname/ceph (note the /ceph path!) to the actual mgr daemon in the (not so) public network of the ceph cluster. In 15.2.9, there would be <base href="https://some.admin.hostname/ceph"> set via JS. This has apparently changed in 15.2.10 -- see PR https://github.com/ceph/ceph/pull/39372 -- leaving a plain <base href="/"> in the HTML header. Browsers (tested with Firefox and Chromium on Debian Buster) now try to load further JS for dashboard functionality from https://some.admin.hostname/ *without* the /ceph path, which obviously fails. Is there any way to configure the path in the <base href=""> tag, e.g. some "ceph dashboard" cli option? The ceph cluster runs non-containerized on Ubuntu 20.04, using the packages from download.ceph.com, and was deployed using ceph-ansible. Best, Christoph -- Dr. Christoph Brüning Universität Würzburg HPC & DataManagement @ ct.qmat & RZUW Am Hubland D-97074 Würzburg Tel.: +49 931 31-80499

1 0

Preferred order of operations when changing crush map and pool rules
by Thomas Hukkelberg 30 Mar '21

30 Mar '21

Hi all! We run a 1.5PB cluster with 12 hosts, 192 OSDs (mix of NVMe and HDD) and need to improve our failure domain by altering the crush rules and moving rack to pods, which would imply a lot of data movement. I wonder what would the preferred order of operations be when doing such changes to the crush map and pools? Will there be minimal data movement by moving all racks to pods at once and change pool repl rules or is the best approach to first move racks one by one to pods and then change pool replication rules from rack to pods? Anyhow I guess it's good practice to set 'norebalance' before moving hosts and unset to start the actual moving? Right now we have the following setup: root -> rack2 -> ups1 + node51 + node57 + switch21 root -> rack3 -> ups2 + node52 + node58 + switch22 root -> rack4 -> ups3 + node53 + node59 + switch23 root -> rack5 -> ups4 + node54 + node60 -- switch 21 ^^ root -> rack6 -> ups5 + node55 + node61 -- switch 22 ^^ root -> rack7 -> ups6 + node56 + node62 -- switch 23 ^^ Note that racks 5-7 are connected to same ToR switches as racks 2-4. Cluster and frontend network are in different VXLANs connected with dual 40GbE. Failure domain for 3x replicated pools are currently by rack, and after adding hosts 57-62 we realized that if one of the switches reboots or fails, replicated PGs located only on those 4 hosts will be unavailable and force pools offline. I guess the best way would instead like to organize the racks in pods like this: root -> pod1 -> rack2 -> ups1 + node51 + node57 root -> pod1 -> rack5 -> ups4 + node54 + node60 -> switch21 root -> pod2 -> rack3 -> ups2 + node52 + node58 root -> pod2 -> rack6 -> ups5 + node55 + node61 -> switch 22 root -> pod3 -> rack4 -> ups3 + node53 + node59 root -> pod3 -> rack7 -> ups6 + node56 + node62 -> switch 23 The reason for this arrangement is that we in the future plan to organize the pods in different buildings. We're running nautilus 14.2.16 and are about to upgrade to Octopus. Should we upgrade to Octopus before crush changes? Any thoughts or insight on how to achieve this with minimal data movement and risk of cluster downtime would be welcome! --thomas -- Thomas Hukkelberg thomas(a)hovedkvarteret.no

3 2

Fwd: ceph-fuse false passed X_OK check
by Alex Taylor 30 Mar '21

30 Mar '21

Hi Patrick, Any updates? Looking forward to your reply :D On Thu, Dec 17, 2020 at 11:39 AM Patrick Donnelly <pdonnell(a)redhat.com> wrote: > > On Wed, Dec 16, 2020 at 5:46 PM Alex Taylor <alexu4993(a)gmail.com> wrote: > > > > Hi Cephers, > > > > I'm using VSCode remote development with a docker server. It worked OK > > but fails to start the debugger after /root mounted by ceph-fuse. The > > log shows that the binary passes access X_OK check but cannot be > > actually executed. see: > > > > ``` > > strace_log: access("/root/.vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7", > > X_OK) = 0 > > > > root@develop:~# ls -alh > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > -rw-r--r-- 1 root root 978 Dec 10 13:06 > > .vscode-server/extensions/ms-vscode.cpptools-1.1.3/debugAdapters/OpenDebugAD7 > > ``` > > > > I also test the access syscall on ext4, xfs and even cephfs kernel > > client, all of them return -EACCES, which is expected (the extension > > will then explicitly call chmod +x). > > > > After some digging in the code, I found it is probably caused by > > https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L5549-L5550. > > So here come two questions: > > 1. Is this a bug or is there any concern I missed? > > I tried reproducing it with the master branch and could not. It might > be due to an older fuse/ceph. I suggest you upgrade! > I tried the master(332a188d9b3c4eb5c5ad2720b7299913c5a772ee) as well and the issue still exists. My test program is: ``` #include <stdio.h> #include <unistd.h> int main() { int r; const char path[] = "test"; r = access(path, F_OK); printf("file exists: %d\n", r); r = access(path, X_OK); printf("file executable: %d\n", r); return 0; } ``` And the test result: ``` # local filesystem: ext4 root@f626800a6e85:~# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:13 test root@f626800a6e85:~# ./a.out file exists: 0 file executable: -1 root@f626800a6e85:~# findmnt -t fuse.ceph-fuse TARGET SOURCE FSTYPE OPTIONS /root/mnt ceph-fuse fuse.ceph-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other root@f626800a6e85:~# cd mnt # ceph-fuse root@f626800a6e85:~/mnt# ls -l test -rw-r--r-- 1 root root 6 Dec 19 06:10 test root@f626800a6e85:~/mnt# ./a.out file exists: 0 file executable: 0 root@f626800a6e85:~/mnt# ./test bash: ./test: Permission denied ``` Again, ceph-fuse says file `test` is executable but in fact it can't be executed. The kernel version I'm testing on is: ``` root@f626800a6e85:~/mnt# uname -ar Linux f626800a6e85 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux ``` Please try the program above and make sure you're running it as root user, thank you. And if the reproduction still fails, please let me know the kernel version. > > 2. It works again with fuse_default_permissions=true, any drawbacks if > > this option is set? > > Correctness (ironically, for you) and performance. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D >

2 1

Rados gateway static website
by Marcel Kuiper 30 Mar '21

30 Mar '21

despite the examples that can be found on the internet I have troubles setting up a static website that serves from a S3 bucket If anyone could point me in the right direction that would be much appreciated Marcel I created an index.html in the bucket sky gm-rc3-jumphost01@ceph/s3cmd (master)$ ./s3cmd info s3://sky/index.html s3://sky/index.html (object): File size: 42046 Last mod: Tue, 30 Mar 2021 13:28:02 GMT MIME type: text/html Storage: STANDARD MD5 sum: 93acaccebb23a18da33ec4294d99ea1a SSE: none Policy: none CORS: none ACL: *anon*: READ ACL: Generic Sky Account: FULL_CONTROL And curl returns gm-rc3-jumphost01@tmp/skills$ curl https://sky.static.gm.core.local/index.html <html> <head><title>404 Not Found</title></head> <body> <h1>404 Not Found</h1> <ul> <li>Code: NoSuchWebsiteConfiguration</li> <li>BucketName: sky</li> <li>RequestId: tx0000000000000000000ba-00606327b8-cca124-rc3-gm</li> <li>HostId: cca124-rc3-gm-rc3</li> COnfig of de rados instance [client.radosgw.rc3-gm] debug_rgw = 20 ms_debug = 1 rgw_zonegroup = rc3 rgw_zone = rc3-gm rgw_enable_static_website = true rgw_enable_apis = s3website rgw expose bucket = true rgw_dns_name = gm-rc3-radosgw.gm.core.local rgw_dns_s3website_name = static.gm.core.local rgw_resolve_cname = true host = gm-rc3-s3web01 keyring = /etc/ceph/ceph.client.radosgw.rc3-gm.keyring log_file = /var/log/ceph/radosgw.log user = ceph rgw_frontends = civetweb port=443s ssl_certificate=/etc/ceph/ssl/key_cert_ca.pem DNS (from pdnsutil list-zone) *.static.gm.core.local 3600 IN CNAME gm-rc3-s3web01.gm.core.local The logs shows 2021-03-30 15:32:53.725 7ff760fcd700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-30 15:32:58.409 7ff746798700 20 HTTP_ACCEPT=*/* 2021-03-30 15:32:58.409 7ff746798700 20 HTTP_HOST=sky.static.gm.core.local 2021-03-30 15:32:58.409 7ff746798700 20 HTTP_USER_AGENT=curl/7.58.0 2021-03-30 15:32:58.409 7ff746798700 20 HTTP_VERSION=1.1 2021-03-30 15:32:58.409 7ff746798700 20 REMOTE_ADDR=10.128.160.47 2021-03-30 15:32:58.409 7ff746798700 20 REQUEST_METHOD=GET 2021-03-30 15:32:58.409 7ff746798700 20 REQUEST_URI=/index.html 2021-03-30 15:32:58.409 7ff746798700 20 SCRIPT_URI=/index.html 2021-03-30 15:32:58.409 7ff746798700 20 SERVER_PORT=443 2021-03-30 15:32:58.409 7ff746798700 20 SERVER_PORT_SECURE=443 2021-03-30 15:32:58.409 7ff746798700 1 ====== starting new request req=0x7ff746791740 ===== 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s initializing for trans_id = tx0000000000000000000c4-006063288a-cca124-rc3-gm 2021-03-30 15:32:58.409 7ff746798700 10 rgw api priority: s3=-1 s3website=1 2021-03-30 15:32:58.409 7ff746798700 10 host=sky.static.gm.core.local 2021-03-30 15:32:58.409 7ff746798700 20 subdomain=sky domain=static.gm.core.local in_hosted_domain=1 in_hosted_domain_s3website=1 2021-03-30 15:32:58.409 7ff746798700 20 final domain/bucket subdomain=sky domain=static.gm.core.local in_hosted_domain=1 in_hosted_domain_s3website=1 s->info.domain=static.gm.core.local s->info.request_uri=/sky/index.html 2021-03-30 15:32:58.409 7ff746798700 20 get_handler handler=29RGWHandler_REST_Obj_S3Website 2021-03-30 15:32:58.409 7ff746798700 10 handler=29RGWHandler_REST_Obj_S3Website 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s getting op 0 2021-03-30 15:32:58.409 7ff746798700 10 op=28RGWGetObj_ObjStore_S3Website 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s s3:get_obj verifying requester 2021-03-30 15:32:58.409 7ff746798700 20 req 196 0.000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2021-03-30 15:32:58.409 7ff746798700 20 req 196 0.000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2021-03-30 15:32:58.409 7ff746798700 20 req 196 0.000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2021-03-30 15:32:58.409 7ff746798700 20 req 196 0.000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s s3:get_obj normalizing buckets and tenants 2021-03-30 15:32:58.409 7ff746798700 10 s->object=index.html s->bucket=sky 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s s3:get_obj init permissions 2021-03-30 15:32:58.409 7ff746798700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>sky</ID><DisplayName>Generic Sky Account</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>sky</ID><DisplayName>Generic Sky Account</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2021-03-30 15:32:58.409 7ff746798700 20 get_system_obj_state: rctx=0x7ff74678f310 obj=rc3-gm.rgw.meta:users.uid:anonymous state=0x55835be11220 s->prefetch_data=0 2021-03-30 15:32:58.409 7ff746798700 10 cache get: name=rc3-gm.rgw.meta+users.uid+anonymous : hit (negative entry) 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s s3:get_obj recalculating target 2021-03-30 15:32:58.409 7ff746798700 10 retarget Starting retarget 2021-03-30 15:32:58.409 7ff746798700 10 RGWHandler_REST_S3Website::error_handler err_no=-2039 http_ret=404 2021-03-30 15:32:58.409 7ff746798700 20 No special error handling today! 2021-03-30 15:32:58.409 7ff746798700 20 op->ERRORHANDLER: err_no=-2039 new_err_no=-2039 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s s3:get_obj op status=0 2021-03-30 15:32:58.409 7ff746798700 2 req 196 0.000s s3:get_obj http status=404 2021-03-30 15:32:58.409 7ff746798700 1 ====== req done req=0x7ff746791740 op status=0 http_status=404 latency=0s ====== 2021-03-30 15:32:58.409 7ff746798700 20 process_request() returned -2039 2021-03-30 15:32:58.409 7ff746798700 1 civetweb: 0x55835b24e000: 10.128.160.47 - - [30/Mar/2021:15:32:58 +0200] "GET /index.html HTTP/1.1" 404 511 - curl/7.58.0

2 2

Upgrade from Luminous to Nautilus now one MDS with could not get service secret
by Robert LeBlanc 30 Mar '21

30 Mar '21

We just upgraded our cluster from Lumious to Nautilus and after a few days one of our MDS servers is getting: 2021-03-28 18:06:32.304 7f57c37ff700 5 mds.beacon.sun-gcs01-mds02 Sending beacon up:standby seq 16 2021-03-28 18:06:32.304 7f57c37ff700 20 mds.beacon.sun-gcs01-mds02 sender thread waiting interval 4s 2021-03-28 18:06:32.308 7f57c8809700 5 mds.beacon.sun-gcs01-mds02 received beacon reply up:standby seq 16 rtt 0.00400001 2021-03-28 18:06:36.308 7f57c37ff700 5 mds.beacon.sun-gcs01-mds02 Sending beacon up:standby seq 17 2021-03-28 18:06:36.308 7f57c37ff700 20 mds.beacon.sun-gcs01-mds02 sender thread waiting interval 4s 2021-03-28 18:06:36.308 7f57c8809700 5 mds.beacon.sun-gcs01-mds02 received beacon reply up:standby seq 17 rtt 0 2021-03-28 18:06:37.788 7f57c900a700 0 auth: could not find secret_id=34586 2021-03-28 18:06:37.788 7f57c900a700 0 cephx: verify_authorizer could not get service secret for service mds secret_id=34586 2021-03-28 18:06:37.788 7f57c6004700 5 mds.sun-gcs01-mds02 ms_handle_reset on v2:10.65.101.13:46566/0 2021-03-28 18:06:40.308 7f57c37ff700 5 mds.beacon.sun-gcs01-mds02 Sending beacon up:standby seq 18 2021-03-28 18:06:40.308 7f57c37ff700 20 mds.beacon.sun-gcs01-mds02 sender thread waiting interval 4s 2021-03-28 18:06:40.308 7f57c8809700 5 mds.beacon.sun-gcs01-mds02 received beacon reply up:standby seq 18 rtt 0 2021-03-28 18:06:44.304 7f57c37ff700 5 mds.beacon.sun-gcs01-mds02 Sending beacon up:standby seq 19 2021-03-28 18:06:44.304 7f57c37ff700 20 mds.beacon.sun-gcs01-mds02 sender thread waiting interval 4s I've tried removing the /var/lib/ceph/mds/ directory and getting the key again. I've removed the key and generated a new one, I've checked the clocks between all the nodes. From what I can tell, everything is good. We did have an issue where the monitor cluster fell over and would not boot. We reduced the monitors to a single monitor, disabled cephx, pulled it off the network and restarted the service a few times which allowed it to come up. We then expanded back to three mons and reenabled cephx and everything has been good until this. No other services seem to be suffering from this and it even appears that the MDS works okay even with these messages. We would like to figure out how to resolve this. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 1

Re: should I increase the amount of PGs?
by Boris Behrens 30 Mar '21

30 Mar '21

One week later the ceph is still balancing. What worries me like hell is the %USE on a lot of those OSDs. Does ceph resolv this on it's own? We are currently down to 5TB space in the cluster. Rebalancing single OSDs doesn't work well and it increases the "missplaced objects". I thought about letting upmap do some rebalancing. Anyone know if this is a good idea? Or if I should bite my nails an wait as I am the headache of my life. [root@s3db1 ~]# ceph osd getmap -o om; osdmaptool om --upmap out.txt --upmap-pool eu-central-1.rgw.buckets.data --upmap-max 10; cat out.txt got osdmap epoch 321975 osdmaptool: osdmap file 'om' writing upmap command output to: out.txt checking for upmap cleanups upmap, max-count 10, max deviation 5 limiting to pools eu-central-1.rgw.buckets.data ([11]) pools eu-central-1.rgw.buckets.data prepared 10/10 changes ceph osd rm-pg-upmap-items 11.209 ceph osd rm-pg-upmap-items 11.253 ceph osd pg-upmap-items 11.7f 79 88 ceph osd pg-upmap-items 11.fc 53 31 105 78 ceph osd pg-upmap-items 11.1d8 84 50 ceph osd pg-upmap-items 11.47f 94 86 ceph osd pg-upmap-items 11.49c 44 71 ceph osd pg-upmap-items 11.553 74 50 ceph osd pg-upmap-items 11.6c3 66 63 ceph osd pg-upmap-items 11.7ad 43 50 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 795.42548 - 795 TiB 626 TiB 587 TiB 82 GiB 1.4 TiB 170 TiB 78.64 1.00 - root default 56 hdd 7.32619 1.00000 7.3 TiB 6.4 TiB 6.4 TiB 684 MiB 16 GiB 910 GiB 87.87 1.12 129 up osd.56 67 hdd 7.27739 1.00000 7.3 TiB 6.4 TiB 6.4 TiB 582 MiB 16 GiB 865 GiB 88.40 1.12 115 up osd.67 79 hdd 3.63689 1.00000 3.6 TiB 3.2 TiB 432 GiB 1.9 GiB 0 B 432 GiB 88.40 1.12 63 up osd.79 53 hdd 7.32619 1.00000 7.3 TiB 6.5 TiB 6.4 TiB 971 MiB 22 GiB 864 GiB 88.48 1.13 114 up osd.53 51 hdd 7.27739 1.00000 7.3 TiB 6.5 TiB 6.4 TiB 734 MiB 15 GiB 837 GiB 88.77 1.13 120 up osd.51 73 hdd 14.55269 1.00000 15 TiB 13 TiB 13 TiB 1.8 GiB 39 GiB 1.6 TiB 88.97 1.13 246 up osd.73 55 hdd 7.32619 1.00000 7.3 TiB 6.5 TiB 6.5 TiB 259 MiB 15 GiB 825 GiB 89.01 1.13 118 up osd.55 70 hdd 7.27739 1.00000 7.3 TiB 6.5 TiB 6.5 TiB 291 MiB 16 GiB 787 GiB 89.44 1.14 119 up osd.70 42 hdd 3.73630 1.00000 3.7 TiB 3.4 TiB 3.3 TiB 685 MiB 8.2 GiB 374 GiB 90.23 1.15 60 up osd.42 94 hdd 3.63869 1.00000 3.6 TiB 3.3 TiB 3.3 TiB 132 MiB 7.7 GiB 345 GiB 90.75 1.15 64 up osd.94 25 hdd 3.73630 1.00000 3.7 TiB 3.4 TiB 3.3 TiB 3.2 MiB 8.1 GiB 352 GiB 90.79 1.15 53 up osd.25 31 hdd 7.32619 1.00000 7.3 TiB 6.7 TiB 6.6 TiB 223 MiB 15 GiB 690 GiB 90.80 1.15 117 up osd.31 84 hdd 7.52150 1.00000 7.5 TiB 6.8 TiB 6.6 TiB 159 MiB 16 GiB 699 GiB 90.93 1.16 121 up osd.84 82 hdd 3.63689 1.00000 3.6 TiB 3.3 TiB 332 GiB 1.0 GiB 0 B 332 GiB 91.08 1.16 59 up osd.82 89 hdd 7.52150 1.00000 7.5 TiB 6.9 TiB 6.6 TiB 400 MiB 15 GiB 670 GiB 91.29 1.16 126 up osd.89 33 hdd 3.73630 1.00000 3.7 TiB 3.4 TiB 3.3 TiB 382 MiB 8.6 GiB 327 GiB 91.46 1.16 66 up osd.33 90 hdd 7.52150 1.00000 7.5 TiB 6.9 TiB 6.6 TiB 338 MiB 15 GiB 658 GiB 91.46 1.16 112 up osd.90 105 hdd 3.63869 0.89999 3.6 TiB 3.3 TiB 3.3 TiB 206 MiB 8.1 GiB 301 GiB 91.91 1.17 56 up osd.105 66 hdd 7.27739 0.95000 7.3 TiB 6.7 TiB 6.7 TiB 322 MiB 16 GiB 548 GiB 92.64 1.18 121 up osd.66 46 hdd 7.27739 1.00000 7.3 TiB 6.8 TiB 6.7 TiB 316 MiB 16 GiB 536 GiB 92.81 1.18 119 up osd.46 Am Di., 23. März 2021 um 19:59 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Good point. Thanks for the hint. I changed it for all OSDs from 5 to 1 > *crossing finger* > > Am Di., 23. März 2021 um 19:45 Uhr schrieb Dan van der Ster < > dan(a)vanderster.com>: > >> I see. When splitting PGs, the OSDs will increase is used space >> temporarily to make room for the new PGs. >> When going from 1024->2048 PGs, that means that half of the objects from >> each PG will be copied to a new PG, and then the previous PGs will have >> those objects deleted. >> >> Make sure osd_max_backfills is set to 1, so that not too many PGs are >> moving concurrently. >> >> >> >> On Tue, Mar 23, 2021, 7:39 PM Boris Behrens <bb(a)kervyn.de> wrote: >> >>> Thank you. >>> Currently I do not have any full OSDs (all <90%) but I keep this in mind. >>> What worries me is the ever increasing %USE metric (it went up from >>> around 72% to 75% in three hours). It looks like there is comming a lot of >>> data (there comes barely new data at the moment), but I think this might >>> have to do with my "let's try to increase the PGs to 2048". I hope that >>> ceph begins to split the old PGs into new ones and removes the old PGs. >>> >>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META >>> AVAIL %USE VAR PGS STATUS TYPE NAME >>> -1 795.42548 - 795 TiB 597 TiB 556 TiB 88 GiB 1.4 TiB >>> 198 TiB 75.12 1.00 - root default >>> >>> Am Di., 23. März 2021 um 19:21 Uhr schrieb Dan van der Ster < >>> dan(a)vanderster.com>: >>> >>>> While you're watching things, if an OSD is getting too close for >>>> comfort to the full ratio, you can temporarily increase it, e.g. >>>> ceph osd set-full-ratio 0.96 >>>> >>>> But don't set that too high -- you can really break an OSD if it gets >>>> 100% full (and then can't delete objects or whatever...) >>>> >>>> -- dan >>>> >>>> On Tue, Mar 23, 2021 at 7:17 PM Boris Behrens <bb(a)kervyn.de> wrote: >>>> > >>>> > Ok, then I will try to reweight the most filled OSDs to .95 and see >>>> if this helps. >>>> > >>>> > Am Di., 23. März 2021 um 19:13 Uhr schrieb Dan van der Ster < >>>> dan(a)vanderster.com>: >>>> >> >>>> >> Data goes to *all* PGs uniformly. >>>> >> Max_avail is limited by the available space on the most full OSD -- >>>> >> you should pay close attention to those and make sure they are moving >>>> >> in the right direction (decreasing!) >>>> >> >>>> >> Another point -- IMHO you should aim to get all PGs active+clean >>>> >> before you add yet another batch of new disks. While there are PGs >>>> >> backfilling, your osdmaps are accumulating on the mons and osds -- >>>> >> this itself will start to use a lot of space, and active+clean is the >>>> >> only way to trim the old maps. >>>> >> >>>> >> -- dan >>>> >> >>>> >> On Tue, Mar 23, 2021 at 7:05 PM Boris Behrens <bb(a)kervyn.de> wrote: >>>> >> > >>>> >> > So, >>>> >> > doing nothing and wait for the ceph to recover? >>>> >> > >>>> >> > In theory there should be enough disk space (more disks arriving >>>> tomorrow), but I fear that there might be an issue, when the backups get >>>> exported over night to this s3. Currently the max_avail lingers around 13TB >>>> and I hope, that the data will go to other PGs than the ones that are >>>> currently on filled OSDs. >>>> >> > >>>> >> > >>>> >> > >>>> >> > Am Di., 23. März 2021 um 18:58 Uhr schrieb Dan van der Ster < >>>> dan(a)vanderster.com>: >>>> >> >> >>>> >> >> Hi, >>>> >> >> >>>> >> >> backfill_toofull is not a bad thing when the cluster is really >>>> full >>>> >> >> like yours. You should expect some of the most full OSDs to >>>> eventually >>>> >> >> start decreasing in usage, as the PGs are moved to the new OSDs. >>>> Those >>>> >> >> backfill_toofull states should then resolve themselves as the OSD >>>> >> >> usage flattens out. >>>> >> >> Keep an eye on the usage of the backfill_full and nearfull OSDs >>>> though >>>> >> >> -- if they do eventually go above the full_ratio (95% by default), >>>> >> >> then writes to those OSDs would stop. >>>> >> >> >>>> >> >> But if on the other hand you're suffering from lots of slow ops or >>>> >> >> anything else visible to your users, then you could try to take >>>> some >>>> >> >> actions to slow down the rebalancing. Just let us know if that's >>>> the >>>> >> >> case and we can see about changing osd_max_backfills, some >>>> weights or >>>> >> >> maybe using the upmap-remapped tool. >>>> >> >> >>>> >> >> -- Dan >>>> >> >> >>>> >> >> On Tue, Mar 23, 2021 at 6:07 PM Boris Behrens <bb(a)kervyn.de> >>>> wrote: >>>> >> >> > >>>> >> >> > Ok, I should have listened to you :) >>>> >> >> > >>>> >> >> > In the last week we added more storage but the issue got worse >>>> instead. >>>> >> >> > Today I realized that the PGs were up to 90GB (bytes column in >>>> ceph pg ls said 95705749636), and the balance kept mentioning the 2048 PGs >>>> for this pool. We were at 72% utilization (ceph osd df tree, first line) >>>> for our cluster and I increased the PGs to 2048. >>>> >> >> > >>>> >> >> > Now I am in a world of trouble. >>>> >> >> > The space in the cluster went down, I am at 45% misplaced >>>> objects, and we already added 20x4TB disks just to not go down completly. >>>> >> >> > >>>> >> >> > The utilization is still going up and the overall free space in >>>> the cluster seems to go down. This is what my ceph status looks like and >>>> now I really need help to get that thing back to normal: >>>> >> >> > [root@s3db1 ~]# ceph status >>>> >> >> > cluster: >>>> >> >> > id: dca79fff-ffd0-58f4-1cff-82a2feea05f4 >>>> >> >> > health: HEALTH_WARN >>>> >> >> > 4 backfillfull osd(s) >>>> >> >> > 17 nearfull osd(s) >>>> >> >> > 37 pool(s) backfillfull >>>> >> >> > 13 large omap objects >>>> >> >> > Low space hindering backfill (add storage if this >>>> doesn't resolve itself): 570 pgs backfill_toofull >>>> >> >> > >>>> >> >> > services: >>>> >> >> > mon: 3 daemons, quorum >>>> ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3 (age 44m) >>>> >> >> > mgr: ceph-mgr2(active, since 15m), standbys: ceph-mgr3, >>>> ceph-mgr1 >>>> >> >> > mds: 3 up:standby >>>> >> >> > osd: 110 osds: 110 up (since 28m), 110 in (since 28m); 1535 >>>> remapped pgs >>>> >> >> > rgw: 3 daemons active (eu-central-1, eu-msg-1, eu-secure-1) >>>> >> >> > >>>> >> >> > task status: >>>> >> >> > >>>> >> >> > data: >>>> >> >> > pools: 37 pools, 4032 pgs >>>> >> >> > objects: 116.23M objects, 182 TiB >>>> >> >> > usage: 589 TiB used, 206 TiB / 795 TiB avail >>>> >> >> > pgs: 160918554/348689415 objects misplaced (46.150%) >>>> >> >> > 2497 active+clean >>>> >> >> > 779 active+remapped+backfill_wait >>>> >> >> > 538 active+remapped+backfill_wait+backfill_toofull >>>> >> >> > 186 active+remapped+backfilling >>>> >> >> > 32 active+remapped+backfill_toofull >>>> >> >> > >>>> >> >> > io: >>>> >> >> > client: 27 MiB/s rd, 69 MiB/s wr, 497 op/s rd, 153 op/s wr >>>> >> >> > recovery: 1.5 GiB/s, 922 objects/s >>>> >> >> > >>>> >> >> > Am Di., 16. März 2021 um 09:34 Uhr schrieb Boris Behrens < >>>> bb(a)kervyn.de>: >>>> >> >> >> >>>> >> >> >> Hi Dan, >>>> >> >> >> >>>> >> >> >> my EC profile look very "default" to me. >>>> >> >> >> [root@s3db1 ~]# ceph osd erasure-code-profile ls >>>> >> >> >> default >>>> >> >> >> [root@s3db1 ~]# ceph osd erasure-code-profile get default >>>> >> >> >> k=2 >>>> >> >> >> m=1 >>>> >> >> >> plugin=jerasure >>>> >> >> >> technique=reed_sol_van >>>> >> >> >> >>>> >> >> >> I don't understand the ouput, but the balancing get worse over >>>> night: >>>> >> >> >> >>>> >> >> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pool-pg-distribution >>>> 11 >>>> >> >> >> Searching for PGs in pools: ['11'] >>>> >> >> >> Summary: 1024 PGs on 84 osds >>>> >> >> >> >>>> >> >> >> Num OSDs with X PGs: >>>> >> >> >> 15: 8 >>>> >> >> >> 16: 7 >>>> >> >> >> 17: 6 >>>> >> >> >> 18: 10 >>>> >> >> >> 19: 1 >>>> >> >> >> 32: 10 >>>> >> >> >> 33: 4 >>>> >> >> >> 34: 6 >>>> >> >> >> 35: 8 >>>> >> >> >> 65: 5 >>>> >> >> >> 66: 5 >>>> >> >> >> 67: 4 >>>> >> >> >> 68: 10 >>>> >> >> >> [root@s3db1 ~]# ceph-scripts/tools/ceph-pg-histogram >>>> --normalize --pool=11 >>>> >> >> >> # NumSamples = 84; Min = 4.12; Max = 5.09 >>>> >> >> >> # Mean = 4.553355; Variance = 0.052415; SD = 0.228942; Median >>>> 4.561608 >>>> >> >> >> # each ∎ represents a count of 1 >>>> >> >> >> 4.1244 - 4.2205 [ 8]: ∎∎∎∎∎∎∎∎ >>>> >> >> >> 4.2205 - 4.3166 [ 6]: ∎∎∎∎∎∎ >>>> >> >> >> 4.3166 - 4.4127 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎ >>>> >> >> >> 4.4127 - 4.5087 [ 10]: ∎∎∎∎∎∎∎∎∎∎ >>>> >> >> >> 4.5087 - 4.6048 [ 11]: ∎∎∎∎∎∎∎∎∎∎∎ >>>> >> >> >> 4.6048 - 4.7009 [ 19]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ >>>> >> >> >> 4.7009 - 4.7970 [ 6]: ∎∎∎∎∎∎ >>>> >> >> >> 4.7970 - 4.8931 [ 8]: ∎∎∎∎∎∎∎∎ >>>> >> >> >> 4.8931 - 4.9892 [ 4]: ∎∎∎∎ >>>> >> >> >> 4.9892 - 5.0852 [ 1]: ∎ >>>> >> >> >> [root@s3db1 ~]# ceph osd df tree | sort -nk 17 | tail >>>> >> >> >> 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 724 GiB 19 GiB >>>> 0 B 724 GiB 80.56 1.07 56 up osd.14 >>>> >> >> >> 19 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 466 MiB >>>> 7.9 GiB 708 GiB 81.25 1.08 53 up osd.19 >>>> >> >> >> 4 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 698 GiB 703 MiB >>>> 0 B 698 GiB 81.27 1.08 48 up osd.4 >>>> >> >> >> 24 hdd 3.63689 1.00000 3.6 TiB 3.0 TiB 695 GiB 640 MiB >>>> 0 B 695 GiB 81.34 1.08 46 up osd.24 >>>> >> >> >> 75 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 440 MiB >>>> 8.1 GiB 704 GiB 81.35 1.08 48 up osd.75 >>>> >> >> >> 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 7.5 MiB >>>> 8.0 GiB 663 GiB 82.44 1.09 47 up osd.71 >>>> >> >> >> 76 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 251 MiB >>>> 9.0 GiB 617 GiB 83.65 1.11 50 up osd.76 >>>> >> >> >> 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 399 MiB >>>> 8.1 GiB 618 GiB 83.85 1.11 55 up osd.33 >>>> >> >> >> 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 317 MiB >>>> 8.8 GiB 617 GiB 83.87 1.11 50 up osd.35 >>>> >> >> >> 34 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 451 MiB >>>> 8.7 GiB 545 GiB 85.75 1.14 54 up osd.34 >>>> >> >> >> >>>> >> >> >> Am Mo., 15. März 2021 um 17:23 Uhr schrieb Dan van der Ster < >>>> dan(a)vanderster.com>: >>>> >> >> >>> >>>> >> >> >>> Hi, >>>> >> >> >>> >>>> >> >> >>> How wide are your EC profiles? If they are really wide, you >>>> might be >>>> >> >> >>> reaching the limits of what is physically possible. Also, I'm >>>> not sure >>>> >> >> >>> that upmap in 14.2.11 is very smart about *improving* >>>> existing upmap >>>> >> >> >>> rules for a given PG, in the case that a PG already has an >>>> upmap-items >>>> >> >> >>> entry but it would help the distribution to add more mapping >>>> pairs to >>>> >> >> >>> that entry. What this means, is that it might sometimes be >>>> useful to >>>> >> >> >>> randomly remove some upmap entries and see if the balancer >>>> does a >>>> >> >> >>> better job when it replaces them. >>>> >> >> >>> >>>> >> >> >>> But before you do that, I re-remembered that looking at the >>>> total PG >>>> >> >> >>> numbers is not useful -- you need to check the PGs per OSD >>>> for the >>>> >> >> >>> eu-central-1.rgw.buckets.data pool only. >>>> >> >> >>> >>>> >> >> >>> We have a couple tools that can help with this: >>>> >> >> >>> >>>> >> >> >>> 1. To see the PGs per OSD for a given pool: >>>> >> >> >>> >>>> https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-pool-pg-dis… >>>> >> >> >>> >>>> >> >> >>> E.g.: ./ceph-pool-pg-distribution 11 # to see the >>>> distribution of >>>> >> >> >>> your eu-central-1.rgw.buckets.data pool. >>>> >> >> >>> >>>> >> >> >>> The output looks like this on my well balanced clusters: >>>> >> >> >>> >>>> >> >> >>> # ceph-scripts/tools/ceph-pool-pg-distribution 15 >>>> >> >> >>> Searching for PGs in pools: ['15'] >>>> >> >> >>> Summary: 256 pgs on 56 osds >>>> >> >> >>> >>>> >> >> >>> Num OSDs with X PGs: >>>> >> >> >>> 13: 16 >>>> >> >> >>> 14: 40 >>>> >> >> >>> >>>> >> >> >>> You should expect a trimodal for your cluster. >>>> >> >> >>> >>>> >> >> >>> 2. You can also use another script from that repo to see the >>>> PGs per >>>> >> >> >>> OSD normalized to crush weight: >>>> >> >> >>> ceph-scripts/tools/ceph-pg-histogram --normalize --pool=15 >>>> >> >> >>> >>>> >> >> >>> This might explain what is going wrong. >>>> >> >> >>> >>>> >> >> >>> Cheers, Dan >>>> >> >> >>> >>>> >> >> >>> >>>> >> >> >>> On Mon, Mar 15, 2021 at 3:04 PM Boris Behrens <bb(a)kervyn.de> >>>> wrote: >>>> >> >> >>> > >>>> >> >> >>> > Absolutly: >>>> >> >> >>> > [root@s3db1 ~]# ceph osd df tree >>>> >> >> >>> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP >>>> META AVAIL %USE VAR PGS STATUS TYPE NAME >>>> >> >> >>> > -1 673.54224 - 674 TiB 496 TiB 468 TiB 97 >>>> GiB 1.2 TiB 177 TiB 73.67 1.00 - root default >>>> >> >> >>> > -2 58.30331 - 58 TiB 42 TiB 38 TiB 9.2 >>>> GiB 99 GiB 16 TiB 72.88 0.99 - host s3db1 >>>> >> >> >>> > 23 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 714 >>>> MiB 25 GiB 3.7 TiB 74.87 1.02 194 up osd.23 >>>> >> >> >>> > 69 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.6 >>>> GiB 40 GiB 3.4 TiB 76.32 1.04 199 up osd.69 >>>> >> >> >>> > 73 hdd 14.55269 1.00000 15 TiB 11 TiB 11 TiB 1.3 >>>> GiB 34 GiB 3.8 TiB 74.15 1.01 203 up osd.73 >>>> >> >> >>> > 79 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.8 >>>> GiB 0 B 1.3 TiB 65.44 0.89 47 up osd.79 >>>> >> >> >>> > 80 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 2.2 >>>> GiB 0 B 1.3 TiB 65.34 0.89 48 up osd.80 >>>> >> >> >>> > 81 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.3 TiB 1.1 >>>> GiB 0 B 1.3 TiB 65.38 0.89 47 up osd.81 >>>> >> >> >>> > 82 hdd 3.63689 1.00000 3.6 TiB 2.5 TiB 1.1 TiB 619 >>>> MiB 0 B 1.1 TiB 68.46 0.93 41 up osd.82 >>>> >> >> >>> > -11 50.94173 - 51 TiB 37 TiB 37 TiB 3.5 >>>> GiB 98 GiB 14 TiB 71.90 0.98 - host s3db10 >>>> >> >> >>> > 63 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 647 >>>> MiB 14 GiB 2.0 TiB 72.72 0.99 94 up osd.63 >>>> >> >> >>> > 64 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.2 TiB 668 >>>> MiB 14 GiB 2.0 TiB 72.23 0.98 93 up osd.64 >>>> >> >> >>> > 65 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 227 >>>> MiB 14 GiB 2.1 TiB 71.16 0.97 100 up osd.65 >>>> >> >> >>> > 66 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 313 >>>> MiB 14 GiB 1.9 TiB 74.25 1.01 92 up osd.66 >>>> >> >> >>> > 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 584 >>>> MiB 14 GiB 2.1 TiB 70.63 0.96 96 up osd.67 >>>> >> >> >>> > 68 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 720 >>>> MiB 14 GiB 2.1 TiB 71.72 0.97 101 up osd.68 >>>> >> >> >>> > 70 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 425 >>>> MiB 14 GiB 2.1 TiB 70.59 0.96 97 up osd.70 >>>> >> >> >>> > -12 50.99052 - 51 TiB 38 TiB 37 TiB 2.1 >>>> GiB 97 GiB 13 TiB 73.77 1.00 - host s3db11 >>>> >> >> >>> > 46 hdd 7.27739 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 229 >>>> MiB 14 GiB 1.7 TiB 77.05 1.05 97 up osd.46 >>>> >> >> >>> > 47 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 159 >>>> MiB 13 GiB 2.2 TiB 70.00 0.95 89 up osd.47 >>>> >> >> >>> > 48 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 279 >>>> MiB 14 GiB 2.1 TiB 71.82 0.97 98 up osd.48 >>>> >> >> >>> > 49 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 276 >>>> MiB 14 GiB 1.8 TiB 74.90 1.02 95 up osd.49 >>>> >> >> >>> > 50 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 336 >>>> MiB 14 GiB 2.0 TiB 72.13 0.98 93 up osd.50 >>>> >> >> >>> > 51 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 728 >>>> MiB 15 GiB 1.6 TiB 77.76 1.06 98 up osd.51 >>>> >> >> >>> > 72 hdd 7.32619 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 147 >>>> MiB 13 GiB 2.0 TiB 72.75 0.99 95 up osd.72 >>>> >> >> >>> > -37 58.55478 - 59 TiB 44 TiB 44 TiB 4.4 >>>> GiB 122 GiB 15 TiB 75.20 1.02 - host s3db12 >>>> >> >> >>> > 19 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 454 >>>> MiB 8.2 GiB 780 GiB 79.35 1.08 53 up osd.19 >>>> >> >> >>> > 71 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 7.1 >>>> MiB 8.0 GiB 734 GiB 80.56 1.09 47 up osd.71 >>>> >> >> >>> > 75 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.9 TiB 439 >>>> MiB 8.2 GiB 777 GiB 79.43 1.08 48 up osd.75 >>>> >> >> >>> > 76 hdd 3.68750 1.00000 3.7 TiB 3.0 TiB 3.0 TiB 241 >>>> MiB 8.9 GiB 688 GiB 81.77 1.11 50 up osd.76 >>>> >> >> >>> > 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 >>>> MiB 30 GiB 3.6 TiB 75.44 1.02 201 up osd.77 >>>> >> >> >>> > 78 hdd 14.60159 1.00000 15 TiB 10 TiB 10 TiB 1015 >>>> MiB 28 GiB 4.2 TiB 71.26 0.97 193 up osd.78 >>>> >> >> >>> > 83 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.4 >>>> GiB 30 GiB 3.8 TiB 73.76 1.00 203 up osd.83 >>>> >> >> >>> > -3 58.49872 - 58 TiB 42 TiB 36 TiB 8.2 >>>> GiB 89 GiB 17 TiB 71.71 0.97 - host s3db2 >>>> >> >> >>> > 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.2 >>>> GiB 37 GiB 3.7 TiB 74.58 1.01 196 up osd.1 >>>> >> >> >>> > 3 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 566 >>>> MiB 0 B 1.3 TiB 64.11 0.87 50 up osd.3 >>>> >> >> >>> > 4 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 771 GiB 695 >>>> MiB 0 B 771 GiB 79.30 1.08 48 up osd.4 >>>> >> >> >>> > 5 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 482 >>>> MiB 0 B 1.2 TiB 66.51 0.90 49 up osd.5 >>>> >> >> >>> > 6 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 1.8 >>>> GiB 0 B 1.3 TiB 64.00 0.87 42 up osd.6 >>>> >> >> >>> > 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 639 >>>> MiB 26 GiB 4.0 TiB 72.44 0.98 192 up osd.7 >>>> >> >> >>> > 74 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 907 >>>> MiB 26 GiB 4.2 TiB 71.32 0.97 193 up osd.74 >>>> >> >> >>> > -4 58.49872 - 58 TiB 43 TiB 36 TiB 34 >>>> GiB 85 GiB 16 TiB 72.69 0.99 - host s3db3 >>>> >> >> >>> > 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 980 >>>> MiB 26 GiB 3.8 TiB 74.36 1.01 203 up osd.2 >>>> >> >> >>> > 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.4 >>>> GiB 33 GiB 3.9 TiB 73.51 1.00 186 up osd.9 >>>> >> >> >>> > 10 hdd 14.65039 1.00000 15 TiB 10 TiB 10 TiB 650 >>>> MiB 26 GiB 4.2 TiB 71.64 0.97 201 up osd.10 >>>> >> >> >>> > 12 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 754 >>>> MiB 0 B 1.3 TiB 64.17 0.87 44 up osd.12 >>>> >> >> >>> > 13 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 813 GiB 2.4 >>>> GiB 0 B 813 GiB 78.17 1.06 58 up osd.13 >>>> >> >> >>> > 14 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 797 GiB 19 >>>> GiB 0 B 797 GiB 78.60 1.07 56 up osd.14 >>>> >> >> >>> > 15 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 2.2 >>>> GiB 0 B 1.3 TiB 63.96 0.87 41 up osd.15 >>>> >> >> >>> > -5 58.49872 - 58 TiB 43 TiB 36 TiB 6.7 >>>> GiB 97 GiB 15 TiB 74.04 1.01 - host s3db4 >>>> >> >> >>> > 11 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 940 >>>> MiB 26 GiB 4.0 TiB 72.49 0.98 196 up osd.11 >>>> >> >> >>> > 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1022 >>>> MiB 26 GiB 3.6 TiB 75.23 1.02 204 up osd.17 >>>> >> >> >>> > 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 945 >>>> MiB 45 GiB 3.8 TiB 74.16 1.01 193 up osd.18 >>>> >> >> >>> > 20 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1020 GiB 596 >>>> MiB 0 B 1020 GiB 72.62 0.99 57 up osd.20 >>>> >> >> >>> > 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 1.9 >>>> GiB 0 B 1023 GiB 72.54 0.98 41 up osd.21 >>>> >> >> >>> > 22 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1023 GiB 797 >>>> MiB 0 B 1023 GiB 72.54 0.98 53 up osd.22 >>>> >> >> >>> > 24 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 766 GiB 618 >>>> MiB 0 B 766 GiB 79.42 1.08 46 up osd.24 >>>> >> >> >>> > -6 58.89636 - 59 TiB 43 TiB 43 TiB 3.0 >>>> GiB 108 GiB 16 TiB 73.40 1.00 - host s3db5 >>>> >> >> >>> > 0 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 92 >>>> MiB 7.2 GiB 1.1 TiB 71.16 0.97 45 up osd.0 >>>> >> >> >>> > 25 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 2.4 >>>> MiB 7.3 GiB 1.1 TiB 71.23 0.97 41 up osd.25 >>>> >> >> >>> > 26 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 181 >>>> MiB 7.6 GiB 935 GiB 75.57 1.03 45 up osd.26 >>>> >> >> >>> > 27 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.6 TiB 5.1 >>>> MiB 7.0 GiB 1.1 TiB 71.20 0.97 47 up osd.27 >>>> >> >> >>> > 28 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 977 >>>> MiB 26 GiB 3.8 TiB 73.85 1.00 197 up osd.28 >>>> >> >> >>> > 29 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 872 >>>> MiB 26 GiB 4.1 TiB 71.98 0.98 196 up osd.29 >>>> >> >> >>> > 30 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 943 >>>> MiB 27 GiB 3.6 TiB 75.51 1.03 202 up osd.30 >>>> >> >> >>> > -7 58.89636 - 59 TiB 44 TiB 43 TiB 13 >>>> GiB 122 GiB 15 TiB 74.97 1.02 - host s3db6 >>>> >> >> >>> > 32 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 27 >>>> MiB 7.6 GiB 940 GiB 75.42 1.02 55 up osd.32 >>>> >> >> >>> > 33 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 376 >>>> MiB 8.2 GiB 691 GiB 81.94 1.11 55 up osd.33 >>>> >> >> >>> > 34 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 450 >>>> MiB 8.5 GiB 620 GiB 83.79 1.14 54 up osd.34 >>>> >> >> >>> > 35 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 316 >>>> MiB 8.4 GiB 690 GiB 81.98 1.11 50 up osd.35 >>>> >> >> >>> > 36 hdd 14.65039 1.00000 15 TiB 11 TiB 10 TiB 489 >>>> MiB 25 GiB 4.1 TiB 71.69 0.97 208 up osd.36 >>>> >> >> >>> > 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 >>>> GiB 38 GiB 4.0 TiB 72.41 0.98 195 up osd.37 >>>> >> >> >>> > 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 >>>> GiB 26 GiB 3.7 TiB 74.88 1.02 204 up osd.38 >>>> >> >> >>> > -8 58.89636 - 59 TiB 44 TiB 43 TiB 3.8 >>>> GiB 111 GiB 15 TiB 74.16 1.01 - host s3db7 >>>> >> >> >>> > 39 hdd 3.73630 1.00000 3.7 TiB 2.8 TiB 2.7 TiB 19 >>>> MiB 7.5 GiB 936 GiB 75.54 1.03 39 up osd.39 >>>> >> >> >>> > 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 >>>> MiB 7.1 GiB 1.1 TiB 69.87 0.95 39 up osd.40 >>>> >> >> >>> > 41 hdd 3.73630 1.00000 3.7 TiB 2.7 TiB 2.7 TiB 219 >>>> MiB 7.6 GiB 1011 GiB 73.57 1.00 55 up osd.41 >>>> >> >> >>> > 42 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 593 >>>> MiB 7.1 GiB 1.1 TiB 70.02 0.95 47 up osd.42 >>>> >> >> >>> > 43 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 500 >>>> MiB 27 GiB 3.7 TiB 74.67 1.01 204 up osd.43 >>>> >> >> >>> > 44 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 >>>> GiB 27 GiB 3.7 TiB 74.62 1.01 193 up osd.44 >>>> >> >> >>> > 45 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 >>>> GiB 29 GiB 3.6 TiB 75.16 1.02 204 up osd.45 >>>> >> >> >>> > -9 51.28331 - 51 TiB 39 TiB 39 TiB 4.9 >>>> GiB 107 GiB 12 TiB 76.50 1.04 - host s3db8 >>>> >> >> >>> > 8 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 474 >>>> MiB 14 GiB 1.7 TiB 76.37 1.04 98 up osd.8 >>>> >> >> >>> > 16 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 783 >>>> MiB 15 GiB 1.6 TiB 78.39 1.06 100 up osd.16 >>>> >> >> >>> > 31 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.6 TiB 441 >>>> MiB 14 GiB 1.6 TiB 77.70 1.05 91 up osd.31 >>>> >> >> >>> > 52 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.5 TiB 939 >>>> MiB 14 GiB 1.7 TiB 76.29 1.04 102 up osd.52 >>>> >> >> >>> > 53 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 848 >>>> MiB 18 GiB 1.9 TiB 74.30 1.01 98 up osd.53 >>>> >> >> >>> > 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 >>>> GiB 16 GiB 1.7 TiB 76.99 1.05 106 up osd.54 >>>> >> >> >>> > 55 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 460 >>>> MiB 15 GiB 1.8 TiB 75.46 1.02 105 up osd.55 >>>> >> >> >>> > -10 51.28331 - 51 TiB 37 TiB 37 TiB 3.8 >>>> GiB 96 GiB 14 TiB 72.77 0.99 - host s3db9 >>>> >> >> >>> > 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 846 >>>> MiB 13 GiB 2.1 TiB 71.16 0.97 104 up osd.56 >>>> >> >> >>> > 57 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 513 >>>> MiB 15 GiB 1.7 TiB 76.53 1.04 96 up osd.57 >>>> >> >> >>> > 58 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 604 >>>> MiB 13 GiB 2.1 TiB 71.23 0.97 98 up osd.58 >>>> >> >> >>> > 59 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 414 >>>> MiB 13 GiB 2.2 TiB 70.03 0.95 88 up osd.59 >>>> >> >> >>> > 60 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 227 >>>> MiB 14 GiB 1.8 TiB 75.54 1.03 97 up osd.60 >>>> >> >> >>> > 61 hdd 7.32619 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 456 >>>> MiB 13 GiB 2.2 TiB 70.01 0.95 95 up osd.61 >>>> >> >> >>> > 62 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.4 TiB 843 >>>> MiB 14 GiB 1.8 TiB 74.93 1.02 110 up osd.62 >>>> >> >> >>> > TOTAL 674 TiB 496 TiB 468 TiB 97 >>>> GiB 1.2 TiB 177 TiB 73.67 >>>> >> >> >>> > MIN/MAX VAR: 0.87/1.14 STDDEV: 4.22 >>>> >> >> >>> > >>>> >> >> >>> > Am Mo., 15. März 2021 um 15:02 Uhr schrieb Dan van der Ster >>>> <dan(a)vanderster.com>: >>>> >> >> >>> >> >>>> >> >> >>> >> OK thanks. Indeed "prepared 0/10 changes" means it thinks >>>> things are balanced. >>>> >> >> >>> >> Could you again share the full ceph osd df tree? >>>> >> >> >>> >> >>>> >> >> >>> >> On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens < >>>> bb(a)kervyn.de> wrote: >>>> >> >> >>> >> > >>>> >> >> >>> >> > Hi Dan, >>>> >> >> >>> >> > >>>> >> >> >>> >> > I've set the autoscaler to warn, but it actually does >>>> not warn for now. So not touching it for now. >>>> >> >> >>> >> > >>>> >> >> >>> >> > this is what the log says in minute intervals: >>>> >> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/active >>>> >> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/sleep_interval >>>> >> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/begin_time >>>> >> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/end_time >>>> >> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/begin_weekday >>>> >> >> >>> >> > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/end_weekday >>>> >> >> >>> >> > 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/pool_ids >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] >>>> Optimize plan auto_2021-03-15_13:51:00 >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/mode >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] >>>> Mode upmap, max misplaced 0.050000 >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] >>>> do_upmap >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/upmap_max_iterations >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config >>>> get_config key: mgr/balancer/upmap_max_deviation >>>> >> >> >>> >> > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] >>>> pools ['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec', >>>> 'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc', >>>> 'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email', >>>> 'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage', 'eu-msg-1.rgw.users.keys', >>>> 'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log', >>>> 'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index', >>>> 'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc', >>>> 'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage', >>>> 'whitespace-again-2021-03-10', 'fra-1.rgw.meta', >>>> 'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email', >>>> 'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid', 'eu-msg-1.rgw.control', >>>> '.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control', >>>> 'fra-1.rgw.log', 'default.rgw.data.root', 'whitespace-again-2021-03-10_3', >>>> 'default.rgw.log', 'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root', >>>> 'default.rgw.users.uid', 'eu-central-1.rgw.control'] >>>> >> >> >>> >> > 2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer] >>>> prepared 0/10 changes >>>> >> >> >>> >> > >>>> >> >> >>> >> > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der >>>> Ster <dan(a)vanderster.com>: >>>> >> >> >>> >> >> >>>> >> >> >>> >> >> I suggest to just disable the autoscaler until your >>>> balancing is understood. >>>> >> >> >>> >> >> >>>> >> >> >>> >> >> What does your active mgr log say (with debug_mgr 4/5), >>>> grep balancer >>>> >> >> >>> >> >> /var/log/ceph/ceph-mgr.*.log >>>> >> >> >>> >> >> >>>> >> >> >>> >> >> -- Dan >>>> >> >> >>> >> >> >>>> >> >> >>> >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens < >>>> bb(a)kervyn.de> wrote: >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > Hi, >>>> >> >> >>> >> >> > this unfortunally did not solve my problem. I still >>>> have some OSDs that fill up to 85% >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > According to the logging, the autoscaler might want >>>> to add more PGs to one Bucken and reduce almost all other buckets to 32. >>>> >> >> >>> >> >> > 2021-03-15 12:19:58.825 7f307f601700 4 >>>> mgr[pg_autoscaler] Pool 'eu-central-1.rgw.buckets.data' root_id -1 using >>>> 0.705080476146 of space, bias 1.0, pg target 1974.22533321 quantized to >>>> 2048 (current 1024) >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > Why the balancing does not happen is still nebulous >>>> to me. >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van >>>> der Ster <dan(a)vanderster.com>: >>>> >> >> >>> >> >> >> >>>> >> >> >>> >> >> >> OK >>>> >> >> >>> >> >> >> Btw, you might need to fail to a new mgr... I'm not >>>> sure if the current active will read that new config. >>>> >> >> >>> >> >> >> >>>> >> >> >>> >> >> >> .. dan >>>> >> >> >>> >> >> >> >>>> >> >> >>> >> >> >> >>>> >> >> >>> >> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens < >>>> bb(a)kervyn.de> wrote: >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> Hi, >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> ok thanks. I just changed the value and rewighted >>>> everything back to 1. Now I let it sync the weekend and check how it will >>>> be on monday. >>>> >> >> >>> >> >> >>> We tried to have the systems total storage balanced >>>> as possible. New systems will be with 8TB disks but for the exiting ones we >>>> added 16TB to offset the 4TB disks and we needed a lot of storage fast, >>>> because of a DC move. If you have any recommendations I would be happy to >>>> hear them. >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> Cheers >>>> >> >> >>> >> >> >>> Boris >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van >>>> der Ster <dan(a)vanderster.com>: >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> Thanks. >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> Decreasing the max deviation to 2 or 1 should help >>>> in your case. This option controls when the balancer stops trying to move >>>> PGs around -- by default it stops when the deviation from the mean is 5. >>>> Yes this is too large IMO -- all of our clusters have this set to 1. >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> And given that you have some OSDs with more than >>>> 200 PGs, you definitely shouldn't increase the num PGs. >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> But anyway with your mixed device sizes it might >>>> be challenging to make a perfectly uniform distribution. Give it a try with >>>> 1 though, and let us know how it goes. >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> .. Dan >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> >>>> >> >> >>> >> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens < >>>> bb(a)kervyn.de> wrote: >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> Hi Dan, >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> upmap_max_deviation is default (5) in our >>>> cluster. Is 1 the recommended deviation? >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> I added the whole ceph osd df tree, (I need to >>>> remove some OSDs and readd them as bluestore with SSD, so 69, 73 and 82 are >>>> a bit off now. I also reweighted to try to get the %USE mitigated). >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> I will increase the mgr debugging to see what is >>>> the problem. >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> [root@s3db1 ~]# ceph osd df tree >>>> >> >> >>> >> >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE >>>> DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME >>>> >> >> >>> >> >> >>>>> -1 673.54224 - 659 TiB 491 TiB 464 >>>> TiB 96 GiB 1.2 TiB 168 TiB 74.57 1.00 - root default >>>> >> >> >>> >> >> >>>>> -2 58.30331 - 44 TiB 22 TiB 17 >>>> TiB 5.7 GiB 38 GiB 22 TiB 49.82 0.67 - host s3db1 >>>> >> >> >>> >> >> >>>>> 23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 >>>> TiB 156 MiB 4.4 GiB 13 TiB 12.50 0.17 101 up osd.23 >>>> >> >> >>> >> >> >>>>> 69 hdd 14.55269 0 0 B 0 B >>>> 0 B 0 B 0 B 0 B 0 0 11 up osd.69 >>>> >> >> >>> >> >> >>>>> 73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 >>>> TiB 6.1 MiB 33 GiB 4.2 TiB 71.15 0.95 107 up osd.73 >>>> >> >> >>> >> >> >>>>> 79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 >>>> GiB 2.0 GiB 0 B 747 GiB 79.94 1.07 52 up osd.79 >>>> >> >> >>> >> >> >>>>> 80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 >>>> TiB 1.9 GiB 0 B 1.0 TiB 71.61 0.96 58 up osd.80 >>>> >> >> >>> >> >> >>>>> 81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 >>>> TiB 1.1 GiB 0 B 1.5 TiB 60.07 0.81 55 up osd.81 >>>> >> >> >>> >> >> >>>>> 82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 >>>> TiB 536 MiB 0 B 1.7 TiB 52.68 0.71 30 up osd.82 >>>> >> >> >>> >> >> >>>>> -11 50.94173 - 51 TiB 38 TiB 38 >>>> TiB 3.7 GiB 100 GiB 13 TiB 74.69 1.00 - host s3db10 >>>> >> >> >>> >> >> >>>>> 63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 >>>> TiB 616 MiB 14 GiB 1.7 TiB 76.04 1.02 92 up osd.63 >>>> >> >> >>> >> >> >>>>> 64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 >>>> TiB 820 MiB 15 GiB 1.8 TiB 75.54 1.01 101 up osd.64 >>>> >> >> >>> >> >> >>>>> 65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 >>>> TiB 109 MiB 14 GiB 2.0 TiB 73.17 0.98 105 up osd.65 >>>> >> >> >>> >> >> >>>>> 66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 >>>> TiB 423 MiB 15 GiB 1.4 TiB 80.38 1.08 98 up osd.66 >>>> >> >> >>> >> >> >>>>> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 >>>> TiB 572 MiB 14 GiB 2.2 TiB 70.10 0.94 100 up osd.67 >>>> >> >> >>> >> >> >>>>> 68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 >>>> TiB 630 MiB 13 GiB 2.0 TiB 72.88 0.98 107 up osd.68 >>>> >> >> >>> >> >> >>>>> 70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 >>>> TiB 648 MiB 14 GiB 1.8 TiB 74.73 1.00 102 up osd.70 >>>> >> >> >>> >> >> >>>>> -12 50.99052 - 51 TiB 39 TiB 39 >>>> TiB 2.9 GiB 99 GiB 12 TiB 77.24 1.04 - host s3db11 >>>> >> >> >>> >> >> >>>>> 46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 >>>> TiB 102 MiB 15 GiB 1.5 TiB 78.91 1.06 97 up osd.46 >>>> >> >> >>> >> >> >>>>> 47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 >>>> TiB 61 MiB 13 GiB 2.1 TiB 71.47 0.96 96 up osd.47 >>>> >> >> >>> >> >> >>>>> 48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 >>>> TiB 853 MiB 15 GiB 1.2 TiB 83.46 1.12 109 up osd.48 >>>> >> >> >>> >> >> >>>>> 49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 >>>> TiB 708 MiB 15 GiB 1.5 TiB 78.96 1.06 98 up osd.49 >>>> >> >> >>> >> >> >>>>> 50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 >>>> TiB 472 MiB 15 GiB 1.4 TiB 80.40 1.08 102 up osd.50 >>>> >> >> >>> >> >> >>>>> 51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 >>>> TiB 729 MiB 15 GiB 1.3 TiB 81.70 1.10 110 up osd.51 >>>> >> >> >>> >> >> >>>>> 72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 >>>> TiB 91 MiB 12 GiB 2.5 TiB 65.82 0.88 89 up osd.72 >>>> >> >> >>> >> >> >>>>> -37 58.55478 - 59 TiB 46 TiB 46 >>>> TiB 5.0 GiB 124 GiB 12 TiB 79.04 1.06 - host s3db12 >>>> >> >> >>> >> >> >>>>> 19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 >>>> TiB 462 MiB 8.2 GiB 559 GiB 85.18 1.14 55 up osd.19 >>>> >> >> >>> >> >> >>>>> 71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 >>>> TiB 3.9 MiB 7.8 GiB 825 GiB 78.14 1.05 50 up osd.71 >>>> >> >> >>> >> >> >>>>> 75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 >>>> TiB 576 MiB 8.3 GiB 555 GiB 85.29 1.14 57 up osd.75 >>>> >> >> >>> >> >> >>>>> 76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 >>>> TiB 239 MiB 9.3 GiB 501 GiB 86.73 1.16 50 up osd.76 >>>> >> >> >>> >> >> >>>>> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 >>>> TiB 880 MiB 30 GiB 3.6 TiB 75.57 1.01 202 up osd.77 >>>> >> >> >>> >> >> >>>>> 78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 >>>> TiB 1.0 GiB 30 GiB 3.4 TiB 76.65 1.03 196 up osd.78 >>>> >> >> >>> >> >> >>>>> 83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 >>>> TiB 1.8 GiB 31 GiB 2.9 TiB 80.04 1.07 223 up osd.83 >>>> >> >> >>> >> >> >>>>> -3 58.49872 - 58 TiB 43 TiB 38 >>>> TiB 8.1 GiB 91 GiB 16 TiB 73.15 0.98 - host s3db2 >>>> >> >> >>> >> >> >>>>> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 3.1 GiB 38 GiB 3.6 TiB 75.52 1.01 194 up osd.1 >>>> >> >> >>> >> >> >>>>> 3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 >>>> TiB 418 MiB 0 B 1.4 TiB 60.94 0.82 52 up osd.3 >>>> >> >> >>> >> >> >>>>> 4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 >>>> GiB 845 MiB 0 B 401 GiB 89.23 1.20 53 up osd.4 >>>> >> >> >>> >> >> >>>>> 5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 >>>> TiB 437 MiB 0 B 1.3 TiB 62.88 0.84 51 up osd.5 >>>> >> >> >>> >> >> >>>>> 6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 >>>> TiB 1.8 GiB 0 B 1.7 TiB 54.51 0.73 47 up osd.6 >>>> >> >> >>> >> >> >>>>> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 493 MiB 26 GiB 3.8 TiB 73.90 0.99 185 up osd.7 >>>> >> >> >>> >> >> >>>>> 74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 1.1 GiB 27 GiB 3.5 TiB 76.27 1.02 208 up osd.74 >>>> >> >> >>> >> >> >>>>> -4 58.49872 - 58 TiB 43 TiB 37 >>>> TiB 33 GiB 86 GiB 15 TiB 74.05 0.99 - host s3db3 >>>> >> >> >>> >> >> >>>>> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 850 MiB 26 GiB 4.0 TiB 72.78 0.98 203 up osd.2 >>>> >> >> >>> >> >> >>>>> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 8.3 GiB 33 GiB 3.6 TiB 75.62 1.01 189 up osd.9 >>>> >> >> >>> >> >> >>>>> 10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 663 MiB 28 GiB 3.5 TiB 76.34 1.02 211 up osd.10 >>>> >> >> >>> >> >> >>>>> 12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 >>>> TiB 633 MiB 0 B 1.2 TiB 66.22 0.89 44 up osd.12 >>>> >> >> >>> >> >> >>>>> 13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 >>>> GiB 2.3 GiB 0 B 720 GiB 80.66 1.08 66 up osd.13 >>>> >> >> >>> >> >> >>>>> 14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 >>>> GiB 18 GiB 0 B 552 GiB 85.18 1.14 60 up osd.14 >>>> >> >> >>> >> >> >>>>> 15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 >>>> TiB 2.1 GiB 0 B 1.7 TiB 53.72 0.72 44 up osd.15 >>>> >> >> >>> >> >> >>>>> -5 58.49872 - 58 TiB 45 TiB 37 >>>> TiB 7.2 GiB 99 GiB 14 TiB 76.37 1.02 - host s3db4 >>>> >> >> >>> >> >> >>>>> 11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 >>>> TiB 897 MiB 28 GiB 2.8 TiB 81.15 1.09 205 up osd.11 >>>> >> >> >>> >> >> >>>>> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 1.2 GiB 27 GiB 3.6 TiB 75.38 1.01 211 up osd.17 >>>> >> >> >>> >> >> >>>>> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 965 MiB 44 GiB 4.0 TiB 72.86 0.98 188 up osd.18 >>>> >> >> >>> >> >> >>>>> 20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 >>>> GiB 529 MiB 0 B 796 GiB 78.63 1.05 66 up osd.20 >>>> >> >> >>> >> >> >>>>> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 >>>> TiB 2.1 GiB 0 B 1.1 TiB 70.32 0.94 47 up osd.21 >>>> >> >> >>> >> >> >>>>> 22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 >>>> GiB 882 MiB 0 B 802 GiB 78.47 1.05 58 up osd.22 >>>> >> >> >>> >> >> >>>>> 24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 >>>> GiB 645 MiB 0 B 856 GiB 77.01 1.03 47 up osd.24 >>>> >> >> >>> >> >> >>>>> -6 58.89636 - 59 TiB 44 TiB 44 >>>> TiB 2.4 GiB 111 GiB 15 TiB 75.22 1.01 - host s3db5 >>>> >> >> >>> >> >> >>>>> 0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 >>>> TiB 70 MiB 6.6 GiB 1.3 TiB 65.00 0.87 48 up osd.0 >>>> >> >> >>> >> >> >>>>> 25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 >>>> TiB 5.3 MiB 6.6 GiB 1.4 TiB 63.86 0.86 41 up osd.25 >>>> >> >> >>> >> >> >>>>> 26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 >>>> TiB 181 MiB 7.6 GiB 862 GiB 77.47 1.04 48 up osd.26 >>>> >> >> >>> >> >> >>>>> 27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 >>>> TiB 7.0 MiB 6.1 GiB 1.5 TiB 61.00 0.82 48 up osd.27 >>>> >> >> >>> >> >> >>>>> 28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 >>>> TiB 937 MiB 30 GiB 2.8 TiB 81.19 1.09 203 up osd.28 >>>> >> >> >>> >> >> >>>>> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 536 MiB 26 GiB 3.8 TiB 73.95 0.99 200 up osd.29 >>>> >> >> >>> >> >> >>>>> 30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 >>>> TiB 744 MiB 28 GiB 3.1 TiB 79.07 1.06 207 up osd.30 >>>> >> >> >>> >> >> >>>>> -7 58.89636 - 59 TiB 44 TiB 44 >>>> TiB 14 GiB 122 GiB 14 TiB 75.41 1.01 - host s3db6 >>>> >> >> >>> >> >> >>>>> 32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 >>>> TiB 16 MiB 8.2 GiB 622 GiB 83.74 1.12 65 up osd.32 >>>> >> >> >>> >> >> >>>>> 33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 >>>> TiB 14 MiB 8.1 GiB 740 GiB 80.67 1.08 52 up osd.33 >>>> >> >> >>> >> >> >>>>> 34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 >>>> TiB 449 MiB 7.7 GiB 877 GiB 77.08 1.03 52 up osd.34 >>>> >> >> >>> >> >> >>>>> 35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 >>>> TiB 133 MiB 7.0 GiB 1.4 TiB 62.18 0.83 42 up osd.35 >>>> >> >> >>> >> >> >>>>> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 544 MiB 26 GiB 4.0 TiB 72.98 0.98 220 up osd.36 >>>> >> >> >>> >> >> >>>>> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 11 GiB 38 GiB 3.6 TiB 75.30 1.01 200 up osd.37 >>>> >> >> >>> >> >> >>>>> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 >>>> TiB 1.2 GiB 28 GiB 3.3 TiB 77.43 1.04 217 up osd.38 >>>> >> >> >>> >> >> >>>>> -8 58.89636 - 59 TiB 47 TiB 46 >>>> TiB 3.9 GiB 116 GiB 12 TiB 78.98 1.06 - host s3db7 >>>> >> >> >>> >> >> >>>>> 39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 >>>> TiB 19 MiB 8.5 GiB 499 GiB 86.96 1.17 43 up osd.39 >>>> >> >> >>> >> >> >>>>> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 >>>> TiB 144 MiB 7.0 GiB 1.2 TiB 68.33 0.92 39 up osd.40 >>>> >> >> >>> >> >> >>>>> 41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 >>>> TiB 218 MiB 7.9 GiB 732 GiB 80.86 1.08 64 up osd.41 >>>> >> >> >>> >> >> >>>>> 42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 >>>> TiB 594 MiB 7.0 GiB 1.2 TiB 67.97 0.91 50 up osd.42 >>>> >> >> >>> >> >> >>>>> 43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 >>>> TiB 564 MiB 28 GiB 2.9 TiB 80.32 1.08 213 up osd.43 >>>> >> >> >>> >> >> >>>>> 44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 >>>> TiB 1.3 GiB 28 GiB 3.1 TiB 78.59 1.05 198 up osd.44 >>>> >> >> >>> >> >> >>>>> 45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 >>>> TiB 1.2 GiB 30 GiB 2.8 TiB 81.05 1.09 214 up osd.45 >>>> >> >> >>> >> >> >>>>> -9 51.28331 - 51 TiB 41 TiB 41 >>>> TiB 4.9 GiB 108 GiB 10 TiB 79.75 1.07 - host s3db8 >>>> >> >> >>> >> >> >>>>> 8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 >>>> TiB 472 MiB 15 GiB 1.5 TiB 79.68 1.07 99 up osd.8 >>>> >> >> >>> >> >> >>>>> 16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 >>>> TiB 785 MiB 15 GiB 1.4 TiB 80.25 1.08 97 up osd.16 >>>> >> >> >>> >> >> >>>>> 31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 >>>> TiB 438 MiB 14 GiB 1.8 TiB 75.36 1.01 87 up osd.31 >>>> >> >> >>> >> >> >>>>> 52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 >>>> TiB 844 MiB 15 GiB 1.6 TiB 78.19 1.05 113 up osd.52 >>>> >> >> >>> >> >> >>>>> 53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 >>>> TiB 792 MiB 18 GiB 1.1 TiB 84.46 1.13 109 up osd.53 >>>> >> >> >>> >> >> >>>>> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 >>>> TiB 959 MiB 15 GiB 1.7 TiB 76.73 1.03 115 up osd.54 >>>> >> >> >>> >> >> >>>>> 55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 >>>> TiB 699 MiB 16 GiB 1.2 TiB 83.56 1.12 122 up osd.55 >>>> >> >> >>> >> >> >>>>> -10 51.28331 - 51 TiB 39 TiB 39 >>>> TiB 4.7 GiB 100 GiB 12 TiB 76.05 1.02 - host s3db9 >>>> >> >> >>> >> >> >>>>> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 >>>> TiB 840 MiB 13 GiB 2.1 TiB 71.06 0.95 105 up osd.56 >>>> >> >> >>> >> >> >>>>> 57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 >>>> TiB 1.0 GiB 16 GiB 1.2 TiB 83.17 1.12 102 up osd.57 >>>> >> >> >>> >> >> >>>>> 58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 >>>> TiB 43 MiB 15 GiB 1.4 TiB 81.56 1.09 105 up osd.58 >>>> >> >> >>> >> >> >>>>> 59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 >>>> TiB 429 MiB 15 GiB 1.4 TiB 80.64 1.08 94 up osd.59 >>>> >> >> >>> >> >> >>>>> 60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 >>>> TiB 226 MiB 14 GiB 2.0 TiB 73.25 0.98 101 up osd.60 >>>> >> >> >>> >> >> >>>>> 61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 >>>> TiB 1.1 GiB 12 GiB 2.5 TiB 65.84 0.88 103 up osd.61 >>>> >> >> >>> >> >> >>>>> 62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 >>>> TiB 1.0 GiB 15 GiB 1.7 TiB 76.83 1.03 126 up osd.62 >>>> >> >> >>> >> >> >>>>> TOTAL 674 TiB 501 TiB 473 >>>> TiB 96 GiB 1.2 TiB 173 TiB 74.57 >>>> >> >> >>> >> >> >>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25 >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan >>>> van der Ster <dan(a)vanderster.com>: >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> No, increasing num PGs won't help substantially. >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> Can you share the entire output of ceph osd df >>>> tree ? >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> Did you already set >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> ceph config set mgr >>>> mgr/balancer/upmap_max_deviation 1 >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> ?? >>>> >> >> >>> >> >> >>>>>> And I recommend debug_mgr 4/5 so you can see >>>> some basic upmap balancer logging. >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> .. Dan >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> >>>> >> >> >>> >> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens < >>>> bb(a)kervyn.de> wrote: >>>> >> >> >>> >> >> >>>>>>> >>>> >> >> >>> >> >> >>>>>>> Hello people, >>>> >> >> >>> >> >> >>>>>>> >>>> >> >> >>> >> >> >>>>>>> I am still struggeling with the balancer >>>> >> >> >>> >> >> >>>>>>> ( >>>> https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html) >>>> >> >> >>> >> >> >>>>>>> Now I've read some more and might think that I >>>> do not have enough PGs. >>>> >> >> >>> >> >> >>>>>>> Currently I have 84OSDs and 1024PGs for the >>>> main pool (3008 total). I >>>> >> >> >>> >> >> >>>>>>> have the autoscaler enabled, but I doesn't tell >>>> me to increase the >>>> >> >> >>> >> >> >>>>>>> PGs. >>>> >> >> >>> >> >> >>>>>>> >>>> >> >> >>> >> >> >>>>>>> What do you think? >>>> >> >> >>> >> >> >>>>>>> >>>> >> >> >>> >> >> >>>>>>> -- >>>> >> >> >>> >> >> >>>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft >>>> sich diesmal abweichend >>>> >> >> >>> >> >> >>>>>>> im groÃƒ¼en Saal. >>>> >> >> >>> >> >> >>>>>>> _______________________________________________ >>>> >> >> >>> >> >> >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> >> >> >>> >> >> >>>>>>> To unsubscribe send an email to >>>> ceph-users-leave(a)ceph.io >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> >>>> >> >> >>> >> >> >>>>> -- >>>> >> >> >>> >> >> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft >>>> sich diesmal abweichend im groÃƒ¼en Saal. >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> >>>> >> >> >>> >> >> >>> -- >>>> >> >> >>> >> >> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich >>>> diesmal abweichend im groÃƒ¼en Saal. >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > >>>> >> >> >>> >> >> > -- >>>> >> >> >>> >> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich >>>> diesmal abweichend im groÃƒ¼en Saal. >>>> >> >> >>> >> > >>>> >> >> >>> >> > >>>> >> >> >>> >> > >>>> >> >> >>> >> > -- >>>> >> >> >>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich >>>> diesmal abweichend im groÃƒ¼en Saal. >>>> >> >> >>> > >>>> >> >> >>> > >>>> >> >> >>> > >>>> >> >> >>> > -- >>>> >> >> >>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal >>>> abweichend im groÃƒ¼en Saal. >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> -- >>>> >> >> >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal >>>> abweichend im groÃƒ¼en Saal. >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > -- >>>> >> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal >>>> abweichend im groÃƒ¼en Saal. >>>> >> > >>>> >> > >>>> >> > >>>> >> > -- >>>> >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal >>>> abweichend im groÃƒ¼en Saal. >>>> > >>>> > >>>> > >>>> > -- >>>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend >>>> im groÃƒ¼en Saal. >>>> >>> >>> >>> -- >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >>> groÃƒ¼en Saal. >>> >> > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groÃƒ¼en Saal. > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 6

forceful remap PGs
by Boris Behrens 30 Mar '21

30 Mar '21

Hi, I have a couple OSDs that currently get a lot of data, and are running towards 95% fillrate. I would like to forcefully remap some PGs (they are around 100GB) to more empty OSDs and drop them from the full OSDs. I know this would lead to degraded objects, but I am not sure how long the cluster will stay in a state where it can allocate objects. OSD.105 grew from around 85% to 92% in the last 4 hours. This is the current state cluster: id: dca79fff-ffd0-58f4-1cff-82a2feea05f4 health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 9 backfillfull osd(s) 19 nearfull osd(s) 37 pool(s) backfillfull BlueFS spillover detected on 1 OSD(s) 13 large omap objects Low space hindering backfill (add storage if this doesn't resolve itself): 248 pgs backfill_toofull Degraded data redundancy: 18115/362288820 objects degraded (0.005%), 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum ceph-s3-mon1,ceph-s3-mon2,ceph-s3-mon3 (age 6d) mgr: ceph-mgr2(active, since 6d), standbys: ceph-mgr3, ceph-mgr1 mds: 3 up:standby osd: 110 osds: 110 up (since 4d), 110 in (since 6d); 324 remapped pgs flags noscrub,nodeep-scrub rgw: 4 daemons active (admin, eu-central-1, eu-msg-1, eu-secure-1) task status: data: pools: 37 pools, 4032 pgs objects: 120.76M objects, 197 TiB usage: 620 TiB used, 176 TiB / 795 TiB avail pgs: 18115/362288820 objects degraded (0.005%) 47144186/362288820 objects misplaced (13.013%) 3708 active+clean 241 active+remapped+backfill_wait+backfill_toofull 63 active+remapped+backfill_wait 11 active+remapped+backfilling 6 active+remapped+backfill_toofull 1 active+remapped+backfilling+forced_backfill 1 active+remapped+forced_backfill+backfill_toofull 1 active+undersized+degraded+remapped+backfilling io: client: 23 MiB/s rd, 252 MiB/s wr, 347 op/s rd, 381 op/s wr recovery: 194 MiB/s, 112 objects/s --- ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 795.42548 - 795 TiB 620 TiB 582 TiB 82 GiB 1.4 TiB 176 TiB 77.90 1.00 - root default 84 hdd 7.52150 1.00000 7.5 TiB 6.8 TiB 6.5 TiB 158 MiB 15 GiB 764 GiB 90.07 1.16 121 up osd.84 79 hdd 3.63689 1.00000 3.6 TiB 3.3 TiB 367 GiB 1.9 GiB 0 B 367 GiB 90.15 1.16 64 up osd.79 70 hdd 7.27739 1.00000 7.3 TiB 6.6 TiB 6.5 TiB 268 MiB 15 GiB 730 GiB 90.20 1.16 121 up osd.70 82 hdd 3.63689 1.00000 3.6 TiB 3.3 TiB 364 GiB 1.1 GiB 0 B 364 GiB 90.23 1.16 59 up osd.82 89 hdd 7.52150 1.00000 7.5 TiB 6.8 TiB 6.6 TiB 395 MiB 16 GiB 735 GiB 90.45 1.16 126 up osd.89 90 hdd 7.52150 1.00000 7.5 TiB 6.8 TiB 6.6 TiB 338 MiB 15 GiB 723 GiB 90.62 1.16 112 up osd.90 33 hdd 3.73630 1.00000 3.7 TiB 3.4 TiB 3.3 TiB 382 MiB 8.6 GiB 358 GiB 90.64 1.16 66 up osd.33 66 hdd 7.27739 0.95000 7.3 TiB 6.7 TiB 6.7 TiB 313 MiB 16 GiB 605 GiB 91.88 1.18 122 up osd.66 46 hdd 7.27739 1.00000 7.3 TiB 6.7 TiB 6.7 TiB 312 MiB 16 GiB 601 GiB 91.93 1.18 119 up osd.46 105 hdd 3.63869 0.89999 3.6 TiB 3.4 TiB 3.4 TiB 206 MiB 8.1 GiB 281 GiB 92.45 1.19 58 up osd.105 -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 4

Device class not deleted/set correctly
by Nico Schottelius 30 Mar '21

30 Mar '21

Hello, follow up from my mail from 2020 [0], it seems that OSDs sometimes have "multiple classes" assigned: [15:47:15] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush rm-device-class osd.4 done removing class of osd(s): 4 [15:47:17] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush rm-device-class osd.4 osd.4 belongs to no class, [15:47:20] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush set-device-class xruk osd.4 set osd(s) 4 to class 'xruk' [15:47:45] server6.place6:/var/lib/ceph/osd/ceph-4# ceph osd crush set-device-class xruk osd.4 osd.4 already set to class xruk. set-device-class item id 4 name 'osd.4' device_class 'xruk': no change. [15:47:47] server6.place6:/var/lib/ceph/osd/ceph-4# /usr/bin/ceph-osd -i 4 --pid-file /var/run/ceph/osd.4.pid -c /etc/ceph/ceph.conf --cluster ceph --setuser ceph --setgroup ceph 2021-03-22 15:48:02.773 7fe2f81e4d80 -1 osd.4 94608 log_to_monitors {default=true} 2021-03-22 15:48:02.777 7fe2f81e4d80 -1 osd.4 94608 mon_cmd_maybe_osd_create fail: 'osd.4 has already bound to class 'xruk', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy [15:48:02] server6.place6:/var/lib/ceph/osd/ceph-4# [15:48:02] server6.place6:/var/lib/ceph/osd/ceph-4# We are running ceph 14.2.9. As written before, it also seems that the affected OSD is peering with OSDs from the wrong class (hdd). Does anyone have a hint on how to fix this? Best regards, Nico [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SFJVJI5XUD7… -- Sustainable and modern Infrastructures by ungleich.ch

3 6

ceph Nautilus lost two disk over night everything hangs
by Rainer Krienke 30 Mar '21

30 Mar '21

Hello, i run a ceph Nautilus cluster with 9 hosts and 144 OSDs. Last night we lost two disks, so two OSDs (67,90) are down. The two disks are on two different hosts. A third ODS on a third host repotrts slow ops. ceph is repairing at the moment. Pools affected are eg these ones: pool 35 'pxa-rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 192082 lfor 0/27841/27845 flags hashpspool,selfmanaged_snaps stripe_width 0 pg_num_min 128 target_size_ratio 0.0001 application rbd pool 36 'pxa-ec' erasure size 6 min_size 5 crush_rule 7 object_hash rjenkins pg_num 512 pgp_num 512 last_change 192177 lfor 0/172580/172578 flags hashpspool,ec_overwrites,selfmanaged_snaps stripe_width 16384 pg_num_min 512 target_size_ratio 0.15 application rbd At the mmoment the proxmox-cluster using storage from the seperate ceph cluster hangs. The ppols with date are erasure coded with the following profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 What I do not understand is why access on the virtualization seem to block. Could that be related to min_size of the pools cause this behaviour? How can I find out if this is true or what else is causing the blocking behaviour seen? This is the current status: health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg incomplete Degraded data redundancy: 42384/130014984 objects degraded (0.033%), 4 pgs degraded, 5 pgs undersized 15 daemons have recently crashed 150 slow ops, oldest one blocked for 15901 sec, daemons [osd.60,osd.67] have slow ops. services: mon: 3 daemons, quorum ceph2,ceph5,ceph8 (age 4h) mgr: ceph2(active, since 7w), standbys: ceph5, ceph8, ceph-admin mds: cephfsrz:1 {0=ceph6=up:active} 2 up:standby osd: 144 osds: 142 up (since 4h), 142 in (since 5h); 6 remapped pgs task status: scrub status: mds.ceph6: idle data: pools: 15 pools, 2632 pgs objects: 21.70M objects, 80 TiB usage: 139 TiB used, 378 TiB / 517 TiB avail pgs: 0.038% pgs not active 42384/130014984 objects degraded (0.033%) 2623 active+clean 3 active+undersized+degraded+remapped+backfilling 3 active+clean+scrubbing+deep 1 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfill_wait 1 remapped+incomplete io: client: 2.2 MiB/s rd, 3.6 MiB/s wr, 8 op/s rd, 179 op/s wr recovery: 51 MiB/s, 12 objects/s Thanks a lot Rainer -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312 PGP: http://www.uni-koblenz.de/~krienke/mypgp.html, Fax: +49261287 1001312

5 10

Resolving LARGE_OMAP_OBJECTS
by Drew Weaver 30 Mar '21

30 Mar '21

Howdy, the dashboard on our cluster keeps showing LARGE_OMAP_OBJECTS. I went through this document https://www.suse.com/support/kb/doc/?id=000019698 I've found that we have a total of 5 buckets, each one is owned by a different user. From what I have read on this issue it seems to flip flop between this is an actual problem that will cause real world issues to "we just raised the limit in the next version". Does anyone have any expertise on whether this is an actual problem or if we should just tune the numbers and how do you determine that? One other quick question: is there a way to add usage information for buckets into mgr for version 14? Thanks, -Drew

3 8

OSD Crash During Deep-Scrub
by Dave Hall 30 Mar '21

30 Mar '21

Hello, A while back, I was having an issue with an OSD repeatedly crashing. I ultimately reweighted it to zero and then marked out 'Out'. Since I found that the logs for thoses crashes match https://tracker.ceph.com/issues/46490 . Since the OSD is in a 'Safe-to-Destroy' state, I'm wondering the best course of action - should I just mark it back in? Or should I destroy and rebuild it. If clearing it in the way I have, in combination with updating to 14.2.16, will prevent it from misbehaving, why go through the trouble of destroying and rebuilding? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

2 1

Re: Nautilus - PG Autoscaler Gobal vs Pool Setting
by Dave Hall 29 Mar '21

29 Mar '21

All, In looking at the options for setting the default pg autoscale option, I notice that there is a global option setting and a per-pool option setting. It seems that the options at the pool level are off, warn, and on. The same, I assume for the global setting. Is there a way to get rid of the per-pool setting and set the pool to honor the global setting? I think I'm looking for 'off, warn, on, or global'. It seems that once the per-pool option is set for all of one's pools, the global value is irrelevant. This also implies that in a circumstance where one would want to temporarily suspend autoscaling it would be required to modify the setting for each pool and then to modify it back afterward. Thoughts? Thanks -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu On Mon, Mar 29, 2021 at 1:44 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Yes the PG autoscalar has a way of reducing PG count way too far. There’s > a claim that it’s better in Pacific, but I tend to recommend disabling it > and calculating / setting pg_num manually. > > > On Mar 29, 2021, at 9:06 AM, Dave Hall <kdhall(a)binghamton.edu> wrote: > > > > Eugen, > > > > I didn't really think my cluster was eating itself, but I also didn't > want > > to be in denial. > > > > Regarding the autoscaler, I really thought that it only went up - I > didn't > > expect that it would decrease the number of PGs. Plus, I thought I had > it > > turned off. I see now that it's off globally but enabled for this > > particular pool. Also, I see that the target PG count is lower than the > > current. > > > > I guess you learn something new every day. > > > > -Dave > > > > -- > > Dave Hall > > Binghamton University > > kdhall(a)binghamton.edu > > 607-760-2328 (Cell) > > 607-777-4641 (Office) > > > > > > On Mon, Mar 29, 2021 at 7:52 AM Eugen Block <eblock(a)nde.ag> wrote: > > > >> Hi, > >> > >> that sounds like the pg_autoscaler is doing its work. Check with: > >> > >> ceph osd pool autoscale-status > >> > >> I don't think ceph is eating itself or that you're losing data. ;-) > >> > >> > >> Zitat von Dave Hall <kdhall(a)binghamton.edu>: > >> > >>> Hello, > >>> > >>> About 3 weeks ago I added a node and increased the number of OSDs in my > >>> cluster from 24 to 32, and then marked one old OSD down because it was > >>> frequently crashing. . > >>> > >>> After adding the new OSDs the PG count jumped fairly dramatically, but > >> ever > >>> since, amidst a continuous low level of rebalancing, the number of PGs > >> has > >>> gradually decreased to less by 25% from it's max value. Although I > don't > >>> have specific notes, my perception is that the current number of PGs is > >>> actually lower than it was before I added OSDs. > >>> > >>> So what's going on here? It is possible to imagine that my cluster is > >>> slowly eating itself, and that I'm about to lose 200TB of data. It's > also > >>> possible to imagine that this is all due to the gradual optimization of > >> the > >>> pools. > >>> > >>> Note that the primary pool is an EC 8 + 2 containing about 124TB. > >>> > >>> Thanks. > >>> > >>> -Dave > >>> > >>> -- > >>> Dave Hall > >>> Binghamton University > >>> kdhall(a)binghamton.edu > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Nautilus - PG count decreasing after adding OSDs
by Dave Hall 29 Mar '21

29 Mar '21

Hello, About 3 weeks ago I added a node and increased the number of OSDs in my cluster from 24 to 32, and then marked one old OSD down because it was frequently crashing. . After adding the new OSDs the PG count jumped fairly dramatically, but ever since, amidst a continuous low level of rebalancing, the number of PGs has gradually decreased to less by 25% from it's max value. Although I don't have specific notes, my perception is that the current number of PGs is actually lower than it was before I added OSDs. So what's going on here? It is possible to imagine that my cluster is slowly eating itself, and that I'm about to lose 200TB of data. It's also possible to imagine that this is all due to the gradual optimization of the pools. Note that the primary pool is an EC 8 + 2 containing about 124TB. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 2

Re: Cluster suspends when Add Mon or stop and start after a while.
by Frank Schilder 29 Mar '21

29 Mar '21

Please use the correct list: ceph-users(a)ceph.io Probably same problem I had. Try reducing mon_sync_max_payload_size=4096 and start a new MON. Should just take a few seconds to boot up. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder Sent: 29 March 2021 16:58:19 To: by morphin; ceph-users(a)ceph.com Subject: Re: [ceph-users] Cluster suspends when Add Mon or stop and start after a while. Probably same problem I had. Try reducing mon_sync_max_payload_size=4096 and start a new MON. Should just take a few seconds to boot up. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: by morphin <morphinwithyou(a)gmail.com> Sent: 28 March 2021 11:25:02 To: ceph-users(a)ceph.com Subject: [ceph-users] Cluster suspends when Add Mon or stop and start after a while. Hello! I have a cluster with Datacenter crushmap (A+B).(9+9 = 18 servers) The cluster started with v12.2.0 Luminous 4 years ago. All these years I upgraded the Cluster Luminous > Mimic > v14.2.16 Nautilus. Now I have a weird issue. When I add a mon or shutdown a while and start it up again, all the cluster suspends, ceph -s do not respond and other two monitors starting election while booting mon is syncing. (logs below) 2021-03-28 00:18:23.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:23.782 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:24.292 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:26.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3919 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:29.782 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7899) init, last seen epoch 7899, mid-election, bumping 2021-03-28 00:18:29.812 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:31.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3951 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:31.872 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.072 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:33.282 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:34.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7901) init, last seen epoch 7901, mid-election, bumping 2021-03-28 00:18:34.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:34.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.072 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:36.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3989 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:36.292 7fe2f2e0f700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:39.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7903) init, last seen epoch 7903, mid-election, bumping 2021-03-28 00:18:39.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:40.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4027 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:41.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.812 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:42.312 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:43.882 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:45.302 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:46.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4062 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:47.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7905) init, last seen epoch 7905, mid-election, bumping 2021-03-28 00:18:47.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:51.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4091 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:52.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7907) init, last seen epoch 7907, mid-election, bumping 2021-03-28 00:18:52.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:52.882 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.082 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:54.302 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:56.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4131 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:57.872 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7909) init, last seen epoch 7909, mid-election, bumping 2021-03-28 00:18:57.902 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:19:21.851 7f9127602700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:25.061 7f912ce0d700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:31.481 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:44.291 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:59.311 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:14.321 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:54.481 7f9128604700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #1158. Immutable memtables: 0. 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.490450) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 5] Flushing memtable with next log file: 1158 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054490507, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 391495, "num_deletes": 0, "total_data_size": 26756127, "memory_usage": 32861536, "flush_reason": "Write Buffer Full"} 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 5] Level-0 flush table #1159: started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054615407, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 1159, "file_size": 14913671, "table_properties": {"data_size": 13733422, "index_size": 200663, "filter_size": 978757, "raw_key_size": 18143237, "raw_average_key_size": 46, "raw_value_size": 7829900, "raw_average_value_size": 20, "num_data_blocks": 3398, "num_entries": 391495, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 5] Level-0 flush table #1159: 14913671 bytes OK 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615451) [db/memtable_list.cc:380] [default] Level-0 commit table #1159 started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615892) [db/memtable_list.cc:428] [default] Level-0 commit table #1159: memtable #1 done 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615900) EVENT_LOG_v1 {"time_micros": 1616880054615897, "job": 5, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615925) [db/db_impl_compaction_flush.cc:201] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 2] max score 1.00 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 5] Try to delete WAL files size 24016022, prev total WAL file size 24798914, number of live WAL files 2. 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 4@0 + 2@6 files to L6, score 1.00 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 0, inputs: [1159(14MB) 1155(25MB) 1149(591KB) 1157(24MB)], [1146(68MB) 1147(53MB)] 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054616197, "job": 6, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [1159, 1155, 1149, 1157], "files_L6": [1146, 1147], "score": 1, "input_data_size": 195345305} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #1160: 547031 keys, 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055675107, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 1160, "file_size": 40744575, "table_properties": {"data_size": 39048586, "index_size": 327543, "filter_size": 1367621, "raw_key_size": 25391768, "raw_average_key_size": 46, "raw_value_size": 30761471, "raw_average_value_size": 56, "num_data_blocks": 6004, "num_entries": 547031, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 6] Compacted 4@0 + 2@6 files to L6 => 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676525) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 184.5 rd, 38.5 wr, level 6, files in(4, 2) out(1) MB in(64.7, 121.6) out(38.9), read-write-amplify(3.5) write-amplify(0.6) OK, records in: 3571913, records dropped: 3024882 output_compression: NoCompression 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676541) EVENT_LOG_v1 {"time_micros": 1616880055676534, "job": 6, "event": "compaction_finished", "compaction_time_micros": 1058962, "compaction_time_cpu_micros": 1047128, "output_level": 6, "num_output_files": 1, "total_output_size": 40744575, "num_input_records": 3571913, "num_output_records": 547031, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055677454, "job": 6, "event": "table_file_deletion", "file_number": 1159} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055678431, "job": 6, "event": "table_file_deletion", "file_number": 1157} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679410, "job": 6, "event": "table_file_deletion", "file_number": 1155} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679506, "job": 6, "event": "table_file_deletion", "file_number": 1149} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055681388, "job": 6, "event": "table_file_deletion", "file_number": 1147} 2021-03-28 00:20:55.681 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055684258, "job": 6, "event": "table_file_deletion", "file_number": 1146} As you can see in the leader mon switchs to election mod and gives these logs until the sync is finished. mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id The booting mon only gives this: mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id The booting mon prepares its *.stt tables and I watched the var/lib/ceph/mon/*/*/*.log file and I saw these logs: !osd_snapremoved_epoch_1_0013c460,�!osd_snapremoved_epoch_1_0013c462.� I guess after all these upgrades Ceph missed some variable or something else. The problem is huge because I can not bootstrap a mon without stopping entire cluster. It takes 30min to boot up... I tried to remove and add new mons but nothing changed. I've a backup service and it takes snapshots and backup to blockstorage and deletes the snapshot. It works for 3 years now and the snapremoved log in th mon log under "var/lib/ceph/mon/*/*/*.log" makes me wonder, what if all these snapshots makes huge bottleneck to the mon bootup? What should I do? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

bucket index and WAL/DB
by Marcelo 29 Mar '21

29 Mar '21

Hello everybody. I searched in several places and I couldn't find any information about what the best bucket index and WAL / DB organization would be. I have several hosts consisting of 12 HDDs and 2 NVMes, and currently one of the NVMes serves as WAL / DB for the 10 OSDs and the other NVMe is partitioned in two, serving as 2 OSDs to serve the S3 index pool. I saw in ceph-ansible a playbook (infrastructure-playbooks / lv-create.yml) that creates a division where we have an OSD living with a journal on the same NVMe. The problem is that in lv-vars.yaml used by lv-create.yml it is said that this only applies to the filestore. Is this correct or can I use this same structure with bluestore? Thank you all, Marcelo.

2 4

Nautilus: Reduce the number of managers
by Dave Hall 29 Mar '21

29 Mar '21

Hello, We are in the process of bringing new hardware online that will allow us to get all of the MGRs, MONs, MDSs, etc. off of our OSD nodes and onto dedicated management nodes. I've created MGRs and MONs on the new nodes, and I found procedures for disabling the MONs from the OSD nodes. Now I'm looking for the correct procedure to remove the MGRs from the OSD nodes. I haven't found any reference to this in the documentation. Is it as simple as stopping and disabling the systemd service/target? Or are there Ceph commands? Do I need to clean up /var/lib/ceph/mgr? Same questions about MDS in the near term, but I haven't searched the docs yet. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

memory consumption by osd
by Tony Liu 29 Mar '21

29 Mar '21

Hi, Here is a snippet from top on a node with 10 OSDs. =========================== MiB Mem : 257280.1 total, 2070.1 free, 31881.7 used, 223328.3 buff/cache MiB Swap: 128000.0 total, 126754.7 free, 1245.3 used. 221608.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30492 167 20 0 4483384 2.9g 16696 S 6.0 1.2 707:05.25 ceph-osd 35396 167 20 0 4444952 2.8g 16468 S 5.0 1.1 815:58.52 ceph-osd 33488 167 20 0 4161872 2.8g 16580 S 4.7 1.1 496:07.94 ceph-osd 36371 167 20 0 4387792 3.0g 16748 S 4.3 1.2 762:37.64 ceph-osd 39185 167 20 0 5108244 3.1g 16576 S 4.0 1.2 998:06.73 ceph-osd 38729 167 20 0 4748292 2.8g 16580 S 3.3 1.1 895:03.67 ceph-osd 34439 167 20 0 4492312 2.8g 16796 S 2.0 1.1 921:55.50 ceph-osd 31473 167 20 0 4314500 2.9g 16684 S 1.3 1.2 680:48.09 ceph-osd 32495 167 20 0 4294196 2.8g 16552 S 1.0 1.1 545:14.53 ceph-osd 37230 167 20 0 4586020 2.7g 16620 S 1.0 1.1 844:12.23 ceph-osd =========================== Does it look OK with 2GB free? I can't tell how that 220GB is used for buffer/cache. Is that used by OSDs? Is it controlled by configuration or auto scaling based on physical memory? Any clarifications would be helpful. Thanks! Tony

5 7

Re: How to clear Health Warning status?
by jinguk.kwon＠ungleich.ch 29 Mar '21

29 Mar '21

Hello there, Thank you for your response. There is no error at syslog, dmesg, or SMART. # ceph health detail HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs osd.29 had 38 reads repaired osd.16 had 17 reads repaired How can i clear this waning ? My ceph is version 14.2.9(clear_shards_repaired is not supported.) /dev/sdh1 on /var/lib/ceph/osd/ceph-16 type xfs (rw,relatime,attr2,inode64,noquota) # cat dmesg | grep sdh [ 12.990728] sd 5:2:3:0: [sdh] 19531825152 512-byte logical blocks: (10.0 TB/9.09 TiB) [ 12.990728] sd 5:2:3:0: [sdh] Write Protect is off [ 12.990728] sd 5:2:3:0: [sdh] Mode Sense: 1f 00 00 08 [ 12.990728] sd 5:2:3:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 13.016616] sdh: sdh1 sdh2 [ 13.017780] sd 5:2:3:0: [sdh] Attached SCSI disk # ceph tell osd.29 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 6.464404, "bytes_per_sec": 166100668.21318716, "iops": 39.60148530320815 } # ceph tell osd.16 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 9.6168945000000008, "bytes_per_sec": 111651617.26584397, "iops": 26.619819942914003 } Thank you > On 26 Mar 2021, at 16:04, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > Did you look at syslog, dmesg, or SMART? Mostly likely the drives are failing. > > >> On Mar 25, 2021, at 9:55 PM, jinguk.kwon(a)ungleich.ch wrote: >> >> Hello there, >> >> Thank you for advanced. >> My ceph is ceph version 14.2.9 >> I have a repair issue too. >> >> ceph health detail >> HEALTH_WARN Too many repaired reads on 2 OSDs >> OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs >> osd.29 had 38 reads repaired >> osd.16 had 17 reads repaired >> >> ~# ceph tell osd.16 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "elapsed_sec": 7.1486738159999996, >> "bytes_per_sec": 150201541.10217974, >> "iops": 35.81083800844663 >> } >> ~# ceph tell osd.29 bench >> { >> "bytes_written": 1073741824, >> "blocksize": 4194304, >> "elapsed_sec": 6.9244327500000002, >> "bytes_per_sec": 155065672.9246161, >> "iops": 36.970537406114602 >> } >> >> But it looks like those osds are ok. how can i clear this warning ? >> >> Best regards >> JG >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Do I need to update ceph.conf and restart each OSD after adding more MONs?
by Tony Liu 29 Mar '21

29 Mar '21

Hi, Do I need to update ceph.conf and restart each OSD after adding more MONs? This is with 15.2.8 deployed by cephadm. When adding MON, "mon_host" should be updated accordingly. Given [1], is that update "the monitor cluster’s centralized configuration database" or "runtime overrides set by an administrator"? [1] https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/#config-sourc… Thanks! Tony

3 6

Re: [ Failed ] Upgrade path for Ceph Ansible from Octopus to Pacific
by Lokendra Rathour 29 Mar '21

29 Mar '21

Request the moderators to approve the same. It is since long and the solution to the issue is not found yet. -Lokendra On Tue, 23 Mar 2021, 10:17 Lokendra Rathour, <lokendrarathour(a)gmail.com> wrote: > Hi Team, > I am trying to upgrade my existing Ceph Cluster (using Ceph-ansible) from > current release Octopus to pacific for which I am using a rolling upgrade. > Facing various issues in getting it done, please note as below and suggest: > > issue 1: when updating the all.yml file with ceph release number to 16 and > and ceph release as Pacific: > > *TASK [ceph-validate : validate ceph_repository_community] > ************************************************************************** > > > > > > > > *task path: > /home/ansible/ceph-ansible/roles/ceph-validate/tasks/main.yml:20Tuesday 23 > March 2021 10:00:09 +0530 (0:00:00.141) 0:01:06.028 *********fatal: > [cephnode1]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'fatal: [cephnode2]: FAILED! => changed=false > msg: ceph_stable_release must be either 'nautilus' or 'octopus'fatal: > [cephnode3]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'* > > *Issue 2: by keeping the ceph_stable_release to octopus and changing the > ceph release number to 16 it gives error as :* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *fatal: [cephnode1 -> cephnode1]: FAILED! => changed=true cmd: - ceph - > --cluster - ceph - osd - require-osd-release - pacific delta: > '0:00:00.478257' end: '2021-03-22 19:54:22.892994' invocation: > module_args: _raw_params: ceph --cluster ceph osd require-osd-release > pacific _uses_shell: false argv: null chdir: null > creates: null executable: null removes: null stdin: null > stdin_add_newline: true strip_empty_ends: true warn: true msg: > non-zero return code rc: 22 start: '2021-03-22 19:54:22.414737' stderr: > |- Invalid command: pacific not in luminous|mimic|nautilus|octopus > osd require-osd-release luminous|mimic|nautilus|octopus > [--yes-i-really-mean-it] : set the minimum allowed OSD release to > participate in the cluster Error EINVAL: invalid command stderr_lines: > <omitted> stdout: '' stdout_lines: <omitted>* > > Problem Statement : > > Not able to upgrade to the upper version of Ceph-ansible from octopus to > Pacific. Please suggest/Support. > > -- > ~ Lokendra > skype: lokendrarathour > > >

1 0

Re: [ Failed ] Upgrade path for Ceph Ansible from Octopus to Pacific
by Lokendra Rathour 29 Mar '21

29 Mar '21

HI Robert, Just checked your email on ceph users list. I will try to look deep into the question. For now i have a qery related to the uphrade itself. Is it possible for you to send any link /documents that you are following to upgrade ceph. I am trying to upgrade ceph cluster using ceph ansible on centos 7 from nautilus to octopus but have many el7 related upgrade issue. And similarly i am trying to do it in centos8 but no luck. I have tried posting my query to community (refer mail.in the trail) but it is yet to be posted after moderators approval. Any support would be appreciated thank you once again for your help. -Lokendra On Tue, 23 Mar 2021, 10:17 Lokendra Rathour, <lokendrarathour(a)gmail.com> wrote: > Hi Team, > I am trying to upgrade my existing Ceph Cluster (using Ceph-ansible) from > current release Octopus to pacific for which I am using a rolling upgrade. > Facing various issues in getting it done, please note as below and suggest: > > issue 1: when updating the all.yml file with ceph release number to 16 and > and ceph release as Pacific: > > *TASK [ceph-validate : validate ceph_repository_community] > ************************************************************************** > > > > > > > > *task path: > /home/ansible/ceph-ansible/roles/ceph-validate/tasks/main.yml:20Tuesday 23 > March 2021 10:00:09 +0530 (0:00:00.141) 0:01:06.028 *********fatal: > [cephnode1]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'fatal: [cephnode2]: FAILED! => changed=false > msg: ceph_stable_release must be either 'nautilus' or 'octopus'fatal: > [cephnode3]: FAILED! => changed=false msg: ceph_stable_release must be > either 'nautilus' or 'octopus'* > > *Issue 2: by keeping the ceph_stable_release to octopus and changing the > ceph release number to 16 it gives error as :* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *fatal: [cephnode1 -> cephnode1]: FAILED! => changed=true cmd: - ceph - > --cluster - ceph - osd - require-osd-release - pacific delta: > '0:00:00.478257' end: '2021-03-22 19:54:22.892994' invocation: > module_args: _raw_params: ceph --cluster ceph osd require-osd-release > pacific _uses_shell: false argv: null chdir: null > creates: null executable: null removes: null stdin: null > stdin_add_newline: true strip_empty_ends: true warn: true msg: > non-zero return code rc: 22 start: '2021-03-22 19:54:22.414737' stderr: > |- Invalid command: pacific not in luminous|mimic|nautilus|octopus > osd require-osd-release luminous|mimic|nautilus|octopus > [--yes-i-really-mean-it] : set the minimum allowed OSD release to > participate in the cluster Error EINVAL: invalid command stderr_lines: > <omitted> stdout: '' stdout_lines: <omitted>* > > Problem Statement : > > Not able to upgrade to the upper version of Ceph-ansible from octopus to > Pacific. Please suggest/Support. > > -- > ~ Lokendra > skype: lokendrarathour > > >

1 0

Cluster suspends when Add Mon or stop and start after a while.
by by morphin 28 Mar '21

28 Mar '21

Hello! I have a cluster with Datacenter crushmap (A+B).(9+9 = 18 servers) The cluster started with v12.2.0 Luminous 4 years ago. All these years I upgraded the Cluster Luminous > Mimic > v14.2.16 Nautilus. Now I have a weird issue. When I add a mon or shutdown a while and start it up again, all the cluster suspends, ceph -s do not respond and other two monitors starting election while booting mon is syncing. (logs below) 2021-03-28 00:18:23.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:23.782 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:24.292 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:26.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3919 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:29.782 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7899) init, last seen epoch 7899, mid-election, bumping 2021-03-28 00:18:29.812 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:31.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3951 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:31.872 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.072 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:32.482 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:33.282 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:34.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7901) init, last seen epoch 7901, mid-election, bumping 2021-03-28 00:18:34.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:34.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.072 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:35.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:36.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 3989 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:36.292 7fe2f2e0f700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:39.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7903) init, last seen epoch 7903, mid-election, bumping 2021-03-28 00:18:39.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:40.872 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4027 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:41.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:41.812 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:42.312 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:43.882 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.082 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:44.492 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:45.302 7fe2ede05700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:46.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4062 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:47.812 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7905) init, last seen epoch 7905, mid-election, bumping 2021-03-28 00:18:47.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:51.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4091 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:52.842 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7907) init, last seen epoch 7907, mid-election, bumping 2021-03-28 00:18:52.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:18:52.882 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.082 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:53.492 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:54.302 7fe2f3610700 1 mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:18:56.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 get_health_metrics reporting 4131 slow ops, oldest is log(1 entries from seq 2031 at 2021-03-28 00:08:41.094522) 2021-03-28 00:18:57.872 7fe2f160c700 1 mon.SRV-SB-1(a)1(electing).elector(7909) init, last seen epoch 7909, mid-election, bumping 2021-03-28 00:18:57.902 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model 2021-03-28 00:19:21.851 7f9127602700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:25.061 7f912ce0d700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:31.481 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:44.291 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:19:59.311 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:14.321 7f912c60c700 1 mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id 2021-03-28 00:20:54.481 7f9128604700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #1158. Immutable memtables: 0. 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.490450) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 5] Flushing memtable with next log file: 1158 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054490507, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 391495, "num_deletes": 0, "total_data_size": 26756127, "memory_usage": 32861536, "flush_reason": "Write Buffer Full"} 2021-03-28 00:20:54.481 7f912de0f700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 5] Level-0 flush table #1159: started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054615407, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 1159, "file_size": 14913671, "table_properties": {"data_size": 13733422, "index_size": 200663, "filter_size": 978757, "raw_key_size": 18143237, "raw_average_key_size": 46, "raw_value_size": 7829900, "raw_average_value_size": 20, "num_data_blocks": 3398, "num_entries": 391495, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 5] Level-0 flush table #1159: 14913671 bytes OK 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615451) [db/memtable_list.cc:380] [default] Level-0 commit table #1159 started 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615892) [db/memtable_list.cc:428] [default] Level-0 commit table #1159: memtable #1 done 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615900) EVENT_LOG_v1 {"time_micros": 1616880054615897, "job": 5, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 0, 0, 0, 0, 0, 2], "immutable_memtables": 0} 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: (Original Log Time 2021/03/28-00:20:54.615925) [db/db_impl_compaction_flush.cc:201] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[4 0 0 0 0 0 2] max score 1.00 2021-03-28 00:20:54.611 7f912de0f700 4 rocksdb: [db/db_impl_files.cc:356] [JOB 5] Try to delete WAL files size 24016022, prev total WAL file size 24798914, number of live WAL files 2. 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 4@0 + 2@6 files to L6, score 1.00 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 0, inputs: [1159(14MB) 1155(25MB) 1149(591KB) 1157(24MB)], [1146(68MB) 1147(53MB)] 2021-03-28 00:20:54.611 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880054616197, "job": 6, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [1159, 1155, 1149, 1157], "files_L6": [1146, 1147], "score": 1, "input_data_size": 195345305} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #1160: 547031 keys, 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055675107, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 1160, "file_size": 40744575, "table_properties": {"data_size": 39048586, "index_size": 327543, "filter_size": 1367621, "raw_key_size": 25391768, "raw_average_key_size": 46, "raw_value_size": 30761471, "raw_average_value_size": 56, "num_data_blocks": 6004, "num_entries": 547031, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 6] Compacted 4@0 + 2@6 files to L6 => 40744575 bytes 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676525) [db/compaction_job.cc:768] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 184.5 rd, 38.5 wr, level 6, files in(4, 2) out(1) MB in(64.7, 121.6) out(38.9), read-write-amplify(3.5) write-amplify(0.6) OK, records in: 3571913, records dropped: 3024882 output_compression: NoCompression 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: (Original Log Time 2021/03/28-00:20:55.676541) EVENT_LOG_v1 {"time_micros": 1616880055676534, "job": 6, "event": "compaction_finished", "compaction_time_micros": 1058962, "compaction_time_cpu_micros": 1047128, "output_level": 6, "num_output_files": 1, "total_output_size": 40744575, "num_input_records": 3571913, "num_output_records": 547031, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055677454, "job": 6, "event": "table_file_deletion", "file_number": 1159} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055678431, "job": 6, "event": "table_file_deletion", "file_number": 1157} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679410, "job": 6, "event": "table_file_deletion", "file_number": 1155} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055679506, "job": 6, "event": "table_file_deletion", "file_number": 1149} 2021-03-28 00:20:55.671 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055681388, "job": 6, "event": "table_file_deletion", "file_number": 1147} 2021-03-28 00:20:55.681 7f912e610700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1616880055684258, "job": 6, "event": "table_file_deletion", "file_number": 1146} As you can see in the leader mon switchs to election mod and gives these logs until the sync is finished. mon.SRV-SB-1@1(electing) e9 failed to get devid for : fallback method has serial ''but no model mon.SRV-SB-1@1(electing) e9 handle_auth_request failed to assign global_id The booting mon only gives this: mon.SRV-SB5@2(synchronizing) e9 handle_auth_request failed to assign global_id The booting mon prepares its *.stt tables and I watched the var/lib/ceph/mon/*/*/*.log file and I saw these logs: !osd_snapremoved_epoch_1_0013c460,�!osd_snapremoved_epoch_1_0013c462.� I guess after all these upgrades Ceph missed some variable or something else. The problem is huge because I can not bootstrap a mon without stopping entire cluster. It takes 30min to boot up... I tried to remove and add new mons but nothing changed. I've a backup service and it takes snapshots and backup to blockstorage and deletes the snapshot. It works for 3 years now and the snapremoved log in th mon log under "var/lib/ceph/mon/*/*/*.log" makes me wonder, what if all these snapshots makes huge bottleneck to the mon bootup? What should I do?

1 0

Can I create 8+2 Erasure coding pool on 5 node?
by by morphin 27 Mar '21

27 Mar '21

Hello. I have 5 node Cluster in A datacenter. Also I have same 5 node in B datacenter. They're gonna be 10 node 8+2 EC cluster for backup but I need to add the 5 node later. I have to sync my S3 data with multisite on the 5 node cluster in A datacenter and move them to the B and add the other 5 node to the same cluster. The question is: Can I create 8+2 ec pool on 5 node cluster and add the 5 node later? How can I rebalance the data after that? Or is there any better solution in my case? what should I do?

5 5

haproxy rewrite for s3 subdomain
by Marc 27 Mar '21

27 Mar '21

Has anyone adapted their haproxy config for rewriting new s3 subdomain urls(?) to old path?

1 0

Cephfs metadata and MDS on same node
by Jesper Lykkegaard Karlsen 26 Mar '21

26 Mar '21

Dear Ceph’ers I am about to upgrade MDS nodes for Cephfs in the Ceph cluster (erasure code 8+3 ) I am administrating. Since they will get plenty of memory and CPU cores, I was wondering if it would be a good idea to move metadata OSDs (NVMe's currently on OSD nodes together with cephfs_data ODS (HDD)) to the MDS nodes? Configured as: 4 x MDS with each a metadata OSD and configured with 4 x replication so each metadata OSD would have a complete copy of metadata. I know MDS, stores al lot of metadata in RAM, but if metadata OSDs were on MDS nodes, would that not bring down latency? Anyway, I am just asking for your opinion on this? Pros and cons or even better somebody who actually have tried this? Best regards, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk<mailto:jelka@mbg.au.dk> Tlf: +45 50906203

4 3

Possible to update from luminous 12.2.8 to nautilus latest?
by Szabo, Istvan (Agoda) 26 Mar '21

26 Mar '21

Hi, Is it possible to do a big jump or needs to go slower to luminous latest, then mimic latest, then nautilus latest? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: How to clear Health Warning status?
by jinguk.kwon＠ungleich.ch 26 Mar '21

26 Mar '21

Hello there, Thank you for advanced. My ceph is ceph version 14.2.9 I have a repair issue too. ceph health detail HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS Too many repaired reads on 2 OSDs osd.29 had 38 reads repaired osd.16 had 17 reads repaired ~# ceph tell osd.16 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 7.1486738159999996, "bytes_per_sec": 150201541.10217974, "iops": 35.81083800844663 } ~# ceph tell osd.29 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 6.9244327500000002, "bytes_per_sec": 155065672.9246161, "iops": 36.970537406114602 } But it looks like those osds are ok. how can i clear this warning ? Best regards JG

1 0

How ceph sees when the pool is getting full?
by Szabo, Istvan (Agoda) 26 Mar '21

26 Mar '21

Hi, I want to set alert on the user's pool before it got's full but in nautilus I still haven't found the way which is the value of their data usage based on ceph detail df? POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR k8s-dbss-w-mdb 16 1.8 TiB 488.61k 2.6 TiB 2.87 45 TiB N/A 1.8 TiB 488.61k 0 B 0 B This output is still confusing :/ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

ceph octopus mysterious OSD crash
by Philip Brown 25 Mar '21

25 Mar '21

I've been banging on my ceph octopus test cluster for a few days now. 8 nodes. each node has 2 SSDs and 8 HDDs. They were all autoprovisioned so that each HDD gets an LVM slice of an SSD as a db partition. service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: rotational: 0 things were going pretty good, until... yesterday.. i noticed TWO of the OSDs were "down". I went to check the logs, with journalctl -u ceph-xxxx(a)osd.xxx all it showed were a bunch of generic debug info, and the fact that it stopped. and various automatic attempts to restart. but no indication of what was wrong, and why the restarts KEEP failing. sample output: systemd[1]: Stopped Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00. systemd[1]: Starting Ceph osd.33 for e51eb2fa-7f82-11eb-94d5-78e3b5148f00... bash[9340]: ceph-e51eb2fa-7f82-11eb-94d5-78e3b5148f00-osd.33-activate bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. bash[9340]: WARNING: The same type, major and minor should not be used for multiple devices. podman[9369]: 2021-03-07 16:00:15.543010794 -0800 PST m=+0.318475882 container create podman[9369]: 2021-03-07 16:00:15.73461926 -0800 PST m=+0.510084288 container init ..... bash[1611473]: --> ceph-volume lvm activate successful for osd ID: 33 podman[1611501]: 2021-03-18 10:23:02.564242824 -0700 PDT m=+1.379793448 container died bash[1611473]: ceph-xx-xx-xx-xx-osd.33 bash[1611473]: WARNING: The same type, major and minor should not be used for multiple devices. (repeat, repeat...) podman[1611615]: 2021-03-18 10:23:03.530992487 -0700 PDT m=+0.333130660 container create .... systemd[1]: Started Ceph osd.33 for xx-xx-xx-xx systemd[1]: ceph-xx-xx-xx-xx(a)osd.33.service: main process exited, code=exited, status=1/FAILURE bash[1611797]: ceph-xx-xx-xx-xx-osd.33-deactivate and eventually it just gives up. smartctl -a doesnt show any errors on the HDD dmesg doesnt show anything. So... what do I do? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

6 21

LVM vs. direct disk acess
by Nico Schottelius 25 Mar '21

25 Mar '21

Good evening, I've seen the shift in ceph to focus more on LVM than on plain (direct) access to disks. I was wondering what the motivation is for that. From my point of view OSD disk layouts never change (they are re-added if they do), so the dynamic approach of LVM is probably not the motivation. LVM also adds another layer of indirection and it seems it would be of disadvantage performance wise as well as added complexity for management. The former is probably only a minor degradation, the latter is something I see more as obstacle for maintenance. At ungleich we are using a custom script [0] to format a disk with two partitions, one for the metadata, one for the rest, which seems to be more simple. I assume there are good reasons not to do as we do, but I was wondering what the practical reasons actually are. Best regards, Nico [0] https://code.ungleich.ch/ungleich-public/ungleich-tools/-/blob/master/ceph/… -- Sustainable and modern Infrastructures by ungleich.ch

4 5

Re: Issues upgrading Ceph from 15.2.8 to 15.2.10
by David Rivera 25 Mar '21

25 Mar '21

Hi Julian, You are most likely running into this same issue: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QNR2XRZPEYK… It is podman 2.2 related. I ran into this using CentOS 8.3 and decided to move to CentOS Stream to be able to upgrade the cluster. David

2 1

Wrong PG placement with custom CRUSH rule
by Davíð Steinn Geirsson 25 Mar '21

25 Mar '21

Hi, I have a small cluster of 3 nodes. Each node has 10 or 11 OSDs, mostly HDDs with a couple of SSDs for faster pools. I am trying to set up an erasure coded pool with m=6 k=6, with each node storing 4 chunks on seperate OSDs. Since this seems not possible with the CLI tooling I have written my own CRUSH rule to achieve this, which looks like this: ``` rule 3host4osd { id 3 type erasure min_size 12 max_size 12 step set_chooseleaf_tries 20 step set_choose_tries 100 step take default class hdd step choose indep 3 type host step choose indep 4 type osd step emit } ``` I've set up my erasure code profile and pool: ``` root@virt02:~# ceph osd pool get rbd_erasure crush_rule crush_rule: 3host4osd root@virt02:~# ceph osd pool get rbd_erasure size size: 12 root@virt02:~# ceph osd pool get rbd_erasure min_size min_size: 7 root@virt02:~# ceph osd pool get rbd_erasure erasure_code_profile erasure_code_profile: default root@virt02:~# ceph osd erasure-code-profile get default crush-device-class= crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=6 m=6 plugin=jerasure technique=reed_sol_van w=8 ``` Based on my understanding of ceph, this should pick 3 hosts, then pick 4 OSDs for each of those hosts. This is *almost* the case. However when testing taking out a host after putting a bunch of data on there, it seems 5 PGs (out of 512) seem to have more than 4 chunks placed on the same host. In all cases it's the same host that gets the extra pieces. When the host is out, I see errors: ``` [WRN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive, 5 pgs down pg 2.87 is down, acting [2147483647,2147483647,2147483647,2147483647,22,2147483647,2147483647,20,16,2147483647,17,18] pg 2.f3 is down, acting [2147483647,22,2147483647,2147483647,23,2147483647,18,17,2147483647,2147483647,2147483647,2147483647] pg 2.100 is down, acting [2147483647,18,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,9,20,22,4] pg 2.141 is down, acting [2147483647,18,2147483647,2147483647,2147483647,2147483647,2147483647,2147483647,20,7,4,22] pg 2.1bb is down, acting [20,2147483647,2147483647,2147483647,18,2147483647,23,17,2147483647,2147483647,2147483647,2147483647] ``` As an example, PG 2.87 (rbd_erasure has pool ID 2 according to `ceph osd lspools`): ``` root@virt02:~# ceph pg 2.87 query [...] "up": [ 2, 0, 6, 5, 22, 1, 8, 20, 16, 14, 17, 18 ], "acting": [ 2, 0, 6, 5, 22, 1, 8, 20, 16, 14, 17, 18 ], [...] ``` OSDs 0, 1, 2, 5, 6, 8 and 14 are all running on the same OSD host. All hosts are running ceph octopus 15.2.9. I've put the output of various diagnostic commands into files accessible over HTTPS here: https://dsg.is/ceph_placement_problem_data/ceph_osd_crush_rule_dump_3host4o… https://dsg.is/ceph_placement_problem_data/ceph_osd_lspools.txt https://dsg.is/ceph_placement_problem_data/ceph_osd_pool_get_rbd_erasure_al… https://dsg.is/ceph_placement_problem_data/ceph_pg_2.87_query.txt https://dsg.is/ceph_placement_problem_data/ceph_pg_dump_all.txt https://dsg.is/ceph_placement_problem_data/ceph_pg_ls.txt Any thoughts or ideas what I'm doing wrong? Kind regards, Davíð

1 0

Issues setting up oidc with keycloak
by Mateusz Kozicki 25 Mar '21

25 Mar '21

Hi I'm currently having a bit of an issue with setting up end user authentication and I would be thankful for any tips I could get. The general scenario is like that; end users are authorised thorough webapp and mobile app thorough keycloak. User has to be able to upload and download data using web interface and mobile app. In order to do that I need to get AssumeRoleWithWebIdentity working. I followed the steps outlined in https://docs.ceph.com/en/latest/radosgw/STS/. Following that guide I was able to get AssumeRole example to work, but not AssumeRoleWithWebIdentity. This is the behaviour I'm getting (logged in aws-cli as TESTER): Username TESTER Full name TestUser Suspended No System No Maximum buckets 1000 Capabilities oidc-provider (*) roles (*) $ aws --endpoint=http://10.10.xx.xx iam list-roles { "Roles": [ { "Path": "/", "RoleName": "S3Access", "RoleId": "d1b84ec1-cceb-4c32-a605-f208b30123e2", "Arn": "arn:aws:iam:::role/S3Access", "CreateDate": "2021-03-24T13:08:20.522Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": [ "arn:aws:iam:::oidc-provider/xxxxx.xxxxnt.com/auth/realms/xxxxnt" ] }, "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Condition": { "StringEquals": { "xxxxx.xxxxnt.com/auth/realms/xxxxnt:app_id": "xxxxnt_xxxx_backend" } } } ] }, "MaxSessionDuration": 3600 } ] } $ aws --endpoint=http://10.10.xx.xx iam list-open-id-connect-providers { "OpenIDConnectProviderList": [ { "Arn": "arn:aws:iam:::oidc-provider/xxxxx.xxxxnt.com/auth/realms/xxxxnt" } ] } $ aws --endpoint=http://10.10.xx.xx iam get-open-id-connect-provider --open-id-connect-provider-arn "arn:aws:iam:::oidc-provider/xxxxx.xxxxnt.com/auth/realms/xxxxnt" { "Url": "https://xxxxx.xxxxnt.com/auth/realms/xxxxnt", "ClientIDList": [ "test_ceph" ], "ThumbprintList": [ "02DC870BD9E72360C090Fxxxxxxxxxxxxxxxxxxx" ], "CreateDate": "2021-03-24T12:26:38.173Z" } $ curl -X POST https://xxxxx.xxxxnt.com/auth/realms/xxxxnt/protocol/openid-connect/token -H "Content-Type: application/x-www-form-urlencoded" -d "username=admin" -d "password=omitted" -d "grant_type=password" -d "client_id=test_ceph" -d "client_secret=d01eafe2-xxxx-xxxx-xxxx-xxxxxx7b7dad" {"access_token":"eyJhbGc.........tTRy1bA","expires_in":300,"refresh_expires_in":1800,"refresh_token":"eyJhbG......RU","token_type":"Bearer","not-before-policy":0,"session_state":"3a57b32a-b17c-4b29-bd68-8ce06b6bd2a8","scope":"email account ..... xxxxnt_xxxx_backend profile"} $ aws --debug --endpoint=http://10.10.xx.xx sts assume-role-with-web-identity --role-arn "arn:aws:iam:::role/S3Access" --role-session-name "test" --web-identity-token "eyJhbGc.........tTRy1bA" ..... 2021-03-25 10:17:45,309 - MainThread - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><RequestId>tx000000000000000000032-00605c5538-2cf3a-pl</RequestId><HostId>2cf3a-pl-default</HostId></Error>'| ..... An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown JWT token returned by keycloak contains fields "iss": "https://xxxxx.xxxxnt.com/auth/realms/flexgent"<https://xxxxx.xxxxnt.com/auth/realms/flexgent>, "aud": "xxxxnt_xxxx_backend", "azp": "test_ceph", Thumbprint was generated using example from ceph documentation (curl from jwks_uri). I'm not really sure what might be wrong, I'll be thankful for any hints - including debugging hints, because so far I'm unable to get useful logs on that. [https://softgent.com/wp-content/uploads/2020/01/Zasob-14.png]<https://www.softgent.com> Softgent Sp. z o.o., Budowlanych 31d, 80-298 Gdansk, POLAND KRS: 0000674406, NIP: 9581679801, REGON: 367090912 www.softgent.com Sąd Rejonowy Gdańsk-Północ w Gdańsku, VII Wydział Gospodarczy Krajowego Rejestru Sądowego KRS 0000674406, Kapitał zakładowy: 25 000,00 zł wpłacony w całości.

1 0

CephFS max_file_size
by Mark Schouten 25 Mar '21

25 Mar '21

Hi, There is a default limit of 1TiB for the max_file_size in CephFS. I altered that to 2TiB, but I now got a request for storing a file up to 7TiB. I'd expect the limit to be there for a reason, but what is the risk of setting that value to say 10TiB? -- Mark Schouten <mark(a)tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208

5 4

cephadm rgw bug with uppercase realm and zone.
by Glen Baars 25 Mar '21

25 Mar '21

Hello Ceph Users, Has anyone come across this error when converting to cephadm? 2021-03-25 20:41:05,616 DEBUG /bin/podman: stderr Error: error getting image "ceph-375dcabe-574f-4002-b322-e7f89cf199e1-rgw.COMPANY.LOCATION.NAS-COMPANY-RK2-CEPH06.pcckdr": repository name must be lowercase It seems that cephadm needs to ensure the hostname / realm / zone is in lowercase when passing through to podman. Any known workarounds? All our realms are uppercase. It seems to work fine on ubuntu 20.04 but not centos 7 ( I assume podman issue ) Thanks Glen This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.

1 0

Pacific release candidate v16.1.0 is out
by Josh Durgin 24 Mar '21

24 Mar '21

There are just a couple remaining issues before the final release. Please test it out and report any bugs. The full release notes are in progress here [0]. Notable Changes --------------- * New ``bluestore_rocksdb_options_annex`` config parameter. Complements ``bluestore_rocksdb_options`` and allows setting rocksdb options without repeating the existing defaults. * The cephfs addes two new CDentry tags, 'I' --> 'i' and 'L' --> 'l', and on-RADOS metadata is no longer backwards compatible after upgraded to Pacific or a later release. * $pid expansion in config paths like ``admin_socket`` will now properly expand to the daemon pid for commands like ``ceph-mds`` or ``ceph-osd``. Previously only ``ceph-fuse``/``rbd-nbd`` expanded ``$pid`` with the actual daemon pid. * The allowable options for some ``radosgw-admin`` commands have been changed. * ``mdlog-list``, ``datalog-list``, ``sync-error-list`` no longer accepts start and end dates, but does accept a single optional start marker. * ``mdlog-trim``, ``datalog-trim``, ``sync-error-trim`` only accept a single marker giving the end of the trimmed range. * Similarly the date ranges and marker ranges have been removed on the RESTful DATALog and MDLog list and trim operations. * ceph-volume: The ``lvm batch`` subcommand received a major rewrite. This closed a number of bugs and improves usability in terms of size specification and calculation, as well as idempotency behaviour and disk replacement process. Please refer to https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/ for more detailed information. * Configuration variables for permitted scrub times have changed. The legal values for ``osd_scrub_begin_hour`` and ``osd_scrub_end_hour`` are 0 - 23. The use of 24 is now illegal. Specifying ``0`` for both values causes every hour to be allowed. The legal vaues for ``osd_scrub_begin_week_day`` and ``osd_scrub_end_week_day`` are 0 - 6. The use of 7 is now illegal. Specifying ``0`` for both values causes every day of the week to be allowed. * Multiple file systems in a single Ceph cluster is now stable. New Ceph clusters enable support for multiple file systems by default. Existing clusters must still set the "enable_multiple" flag on the fs. Please see the CephFS documentation for more information. * volume/nfs: Recently "ganesha-" prefix from cluster id and nfs-ganesha common config object was removed, to ensure consistent namespace across different orchestrator backends. Please delete any existing nfs-ganesha clusters prior to upgrading and redeploy new clusters after upgrading to Pacific. * A new health check, DAEMON_OLD_VERSION, will warn if different versions of Ceph are running on daemons. It will generate a health error if multiple versions are detected. This condition must exist for over mon_warn_older_version_delay (set to 1 week by default) in order for the health condition to be triggered. This allows most upgrades to proceed without falsely seeing the warning. If upgrade is paused for an extended time period, health mute can be used like this "ceph health mute DAEMON_OLD_VERSION --sticky". In this case after upgrade has finished use "ceph health unmute DAEMON_OLD_VERSION". * MGR: progress module can now be turned on/off, using the commands: ``ceph progress on`` and ``ceph progress off``. * An AWS-compliant API: "GetTopicAttributes" was added to replace the existing "GetTopic" API. The new API should be used to fetch information about topics used for bucket notifications. * librbd: The shared, read-only parent cache's config option ``immutable_object_cache_watermark`` now has been updated to property reflect the upper cache utilization before space is reclaimed. The default ``immutable_object_cache_watermark`` now is ``0.9``. If the capacity reaches 90% the daemon will delete cold cache. * OSD: the option ``osd_fast_shutdown_notify_mon`` has been introduced to allow the OSD to notify the monitor it is shutting down even if ``osd_fast_shutdown`` is enabled. This helps with the monitor logs on larger clusters, that may get many 'osd.X reported immediately failed by osd.Y' messages, and confuse tools. [0] https://github.com/ceph/ceph/pull/40265

1 0

add and start OSD without rebalancing
by Boris Behrens 24 Mar '21

24 Mar '21

Hi people, I currently try to add ~30 OSDs to our cluster and wanted to use the gentle-rerweight script for that. I use ceph-colume lvm prepare --data /dev/sdX to create the osd and want to start it without weighting it in. systemctl start ceph-osd@OSD starts the OSD with full weight. Is this possible? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 7

Question about MDS cluster's behavior when crash occurs
by 조규진 24 Mar '21

24 Mar '21

Hi all. I have some questions about how MDS cluster works when crash or operation failure occur. 1. I read ceph documentation and code that each MDS has its own journal and some directory operations like rename use distributed transaction mechanism with ceph-defined events (e.g. EPeerUpdate). What I wonder is when one of MDSs is crashed and then reboots, if some metadata in other MDS's journal is not flushed yet, is rebooted MDS read stale metadata? 2. While rename operation is performing on 2 MDSs and one of them is crashed, Does the other MDS rollback modified in-memory metadata ? Thanks, kyujin

1 0

Issues upgrading Ceph from 15.2.8 to 15.2.10
by Julian Fölsch 24 Mar '21

24 Mar '21

Hello, Our podman backed cluster currently is on version 15.2.8, the ceph and cephadm packages on the hosts are 15.2.10. When we check the package version with 'ceph orch upgrade check --ceph-version 15.2.10' it tells us that it failed to pull the image on a host. Inspecting the log we see that it pulls the image successfully but then fails inspecting it[0]. I can run the command manually without problems when I remove the 'json ' from it. So far I was unable to find the issue myself, maybe someone here knows why this happens and how to fix it. Kind regards, Julian Fölsch [0] https://agdsn.me/~paktosan/ceph/octopus_upgrade/cephadm_log.txt -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

1 0

Re: Issue about rbd image(disable feature journaling failed)
by Jason Dillaman 24 Mar '21

24 Mar '21

"rbd status" will tell you which IP addresses have the image open. "rbd lock ls" will show you who owns the lock. On Wed, Mar 24, 2021 at 9:07 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: > > Hi, > As info mentioned in the log file > There still a lock owner alive. The question is how to find out the alive owner and what is the root cause about this. Why cannot acquire lock to the owner? > Thank you so much > > Jason Dillaman <jdillama(a)redhat.com>于2021年3月24日周三20:55写道： >> >> It sounds like this is a non-primary mirrored image, which means it's >> read-only and cannot be modified. A quick "rbd info" will tell you the >> mirror state. Instead, you would need to force-promote it to primary >> via "rbd mirror image promote --force" before attempting to modify the >> image. >> >> On Wed, Mar 24, 2021 at 3:44 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: >> > >> > hi , dillaman: >> > i was confused nowadays, when i execute the rbd feature disable <image> journaling, >> > the result of this often shows me : >> > rbd: failed to update image features: (30) Read-only file system >> > i use the rbd-mirror for replicate data between two clouster. i thought the rbd-mirror could be the reason. but i have no idea about it. could you give some advise ? >> > some log when execute rbd command also send you. >> >> >> >> -- >> Jason >> > -- > 发自移动版 Gmail -- Jason

1 0

Re: Issue about rbd image(disable feature journaling failed)
by Jason Dillaman 24 Mar '21

24 Mar '21

That means you have another librbd client that "owns" the image that you will need to shut down before you can make that change. On Wed, Mar 24, 2021 at 9:01 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: > > hi， > Thank you for the clarification. > It’s already promoted and not mirrored ,a primary image > I have run rbd —debug-rbd=30 and collect a log file > Which shows locker owner still alive and unable get the lock, return -EAGAIN > I’ll send you log later > > Thank you so much > > > Jason Dillaman <jdillama(a)redhat.com>于2021年3月24日周三20:55写道： >> >> It sounds like this is a non-primary mirrored image, which means it's >> read-only and cannot be modified. A quick "rbd info" will tell you the >> mirror state. Instead, you would need to force-promote it to primary >> via "rbd mirror image promote --force" before attempting to modify the >> image. >> >> On Wed, Mar 24, 2021 at 3:44 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: >> > >> > hi , dillaman: >> > i was confused nowadays, when i execute the rbd feature disable <image> journaling, >> > the result of this often shows me : >> > rbd: failed to update image features: (30) Read-only file system >> > i use the rbd-mirror for replicate data between two clouster. i thought the rbd-mirror could be the reason. but i have no idea about it. could you give some advise ? >> > some log when execute rbd command also send you. >> >> >> >> -- >> Jason >> > -- > 发自移动版 Gmail -- Jason

1 0

Re: Issue about rbd image(disable feature journaling failed)
by Jason Dillaman 24 Mar '21

24 Mar '21

It sounds like this is a non-primary mirrored image, which means it's read-only and cannot be modified. A quick "rbd info" will tell you the mirror state. Instead, you would need to force-promote it to primary via "rbd mirror image promote --force" before attempting to modify the image. On Wed, Mar 24, 2021 at 3:44 AM Edgelong Voodu <1070443499cs(a)gmail.com> wrote: > > hi , dillaman: > i was confused nowadays, when i execute the rbd feature disable <image> journaling, > the result of this often shows me : > rbd: failed to update image features: (30) Read-only file system > i use the rbd-mirror for replicate data between two clouster. i thought the rbd-mirror could be the reason. but i have no idea about it. could you give some advise ? > some log when execute rbd command also send you. -- Jason

1 0

should I increase the amount of PGs?
by Boris Behrens 24 Mar '21

24 Mar '21

Hello people, I am still struggeling with the balancer (https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html) Now I've read some more and might think that I do not have enough PGs. Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I have the autoscaler enabled, but I doesn't tell me to increase the PGs. What do you think? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 19

fixing future rctimes
by Dan van der Ster 23 Mar '21

23 Mar '21

Hi all, We have a few subdirs with an rctime in the future. # getfattr -n ceph.dir.rctime session # file: session ceph.dir.rctime="2576387188.090" I can't find any subdir or item in that directory with that rctime, so I presume that there was previously a file and that rctime cannot go backwards [1] Is there any way to fix these rctimes so they show the latest ctime of the subtree? Also -- are we still relying on the client clock to set the rctime / ctime of a file? Would it make sense to limit ctime/rctime for any update to the current time on the MDS ? Best Regards, Dan [1] https://github.com/ceph/ceph/pull/24023/commits/920ef964311a61fcc6c0d6671b7…

2 3

Advice needed: stuck cluster halfway upgraded, comms issues and MON space usage
by Sam Skipsey 23 Mar '21

23 Mar '21

Hi everyone: I posted to the list on Friday morning (UK time), but apparently my email is still in moderation (I have an email from the list bot telling me that it's held for moderation but no updates). Since this is a bit urgent - we have ~3PB of storage offline - I'm posting again. To save retyping the whole thing, I will direct you to a copy of the email I wrote on Friday: http://aoanla.pythonanywhere.com/Logs/EmailToCephUsers.txt (Since that was sent, we did successfully add big SSDs to the MON hosts so they don't fill up their disks with store.db s). I would appreciate any advice - assuming this also doesn't get stuck in moderation queues. -- Sam Skipsey (he/him, they/them)

3 18

Nautilus block-db resize - ceph-bluestore-tool
by Dave Hall 23 Mar '21

23 Mar '21

Hello, Based on other discussions in this list I have concluded that I need to add NVMe to my OSD nodes and expand the NVMe (DB/WAL) for each OSD. Is there a way to do this without destroying and rebuilding each OSD (after safe removal from the cluster, of course)? Is there a way to use ceph-bluestore-tool for this? Is it as simple as lvextend? Why more NVMe? Frequent DB spillovers, and the recommendation that the NVMe should be 40GB for every TB of HDD. When I did my initial setup I thought that 124GB of NVMe for a 12TB HDD would be sufficient, but by the above metric it should be more like 480GB of NVMe. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

Re: How to reset and configure replication on multiple RGW servers from scratch?
by Scheurer François 23 Mar '21

23 Mar '21

Dear All We have the same question here, if anyone can help ... Thank you! We did not find any documentation about the steps to reset & restart the sync. Especially the implications of 'bilog trim', 'mdlog trim' and 'datalog trim'. Our secondary zone is read-only. Both master and secondary zone on Nautilus (master 14.2.9 and secondary 14.2.12). Can someone also clarify following points? Many thanks in advance! 1) Is it safe to use these 3 commands (bilog trim, mdlog trim, datalog trim) on the master ? Are the bi logs exclusively used for the sync or are they needed even without multi-site? (mdlog/datalog are obviously only for multi-site) 2) Can we run these 3 commands during the sync or do we need first to stop all instances on the secondary zone ? In the latter case, do we need to stop the client traffic and wait on md/data sync to catch up prior to stop the secondary zone instances? 3) Can we then restart the instances on the secondary zone and expect rgw sync to run correctly ? Or do we need first to run 'metadata sync init' and 'data sync init' on the secondary zone ? (to trigger a full sync) Or is it necessary to delete all rgw pools on the secondary zone ? 4) And regarding the full sync, is it verifying the full object data, or only object size and mtime? If we update the secondary zone to Nautilus 14.2.18 and enable rgw_sync_obj_etag_verify, does a full sync will also detect ETag mismatches on objects that are already present on the secondary zone? Cheers Francois ________________________________ From: ceph-users on behalf of Osiński Piotr <Piotr.Osinski(a)grupawp.pl> Sent: Saturday, June 22, 2019 11:44 AM To: ceph-users(a)lists.ceph.com Subject: [ceph-users] How to reset and configure replication on multiple RGW servers from scratch? Hi, For testing purposes, I configured RGW multisite synchronization between two ceph mimic 13.2.6 clusters (I also tried: 13.2.5). Now I want to reset all current settings and configure replication from scratch. Data(pools, buckets) on the master zone will not be deleted. What has been done: 1) Deleted the secondary zone # radosgw-admin zone delete --rgw-zone=dc2_zone 2) Removed the secondary zone from zonegroup # radosgw-admin zonegroup remove --rgw-zonegroup=master_zonegroup --rgw-zone=dc2_zone 3) Commited changes # radosgw-admin period update --commit 4) Trimmed all datalogs on master zone # radosgw-admin datalog trim --start-date="2019-06-12 12:01:54" --end-date="2019-06-22 12:01:56" 5) Trimmed all error sync on master zone # radosgw-admin sync error trim --start-date="2019-06-07 07:19:26" --end-date="2019-06-22 15:59:00" 6) Deleted and recreated empty pools on secondary cluster: dc2_zone.rgw.control dc2_zone.rgw.meta dc2_zone.rgw.log dc2_zone.rgw.buckets.index dc2_zone.rgw.buckets.data Should I clear any other data / metadata in the master zone? Can data be kept somewhere in the master zone that may affect the new replication statement? I'm trying to track down a problem with blocked shards synchronization. Thank you in advance for your help. Best regards, Piotr Osiński << ATT00001.txt (0.4KB) (0.4KB) >>

1 0

New Issue - Mapping Block Devices
by duluxoz 23 Mar '21

23 Mar '21

Hi All, I've got a new issue (hopefully this one will be the last). I have a working Ceph (Octopus) cluster with a replicated pool (my-pool), an erasure-coded pool (my-pool-data), and an image (my-image) created - all *seems* to be working correctly. I also have the correct Keyring specified (ceph.client.my-id.keyring). ceph -s is reporting all healthy. The ec profile (my-ec-profile) was created with: ceph osd erasure-code-profile set my-ec-profile k=4 m=2 crush-failure-domain=host The replicated pool was created with: ceph osd pool create my-pool 100 100 replicated Followed by: rbd pool init my-pool The ec pool was created with: ceph osd pool create my-pool-data 100 100 erasure my-ec-profile --autoscale-mode=on Followed by: rbd pool init my-pool-data The image was created with: rbd create -s 1T --data-pool my-pool-data my-pool/my-image The Keyring was created with: ceph auth get-or-create client.my-id mon 'profile rbd' osd 'profile rbd pool=my-pool' mgr 'profile rbd pool=my-pool' -o /etc/ceph/ceph.client.my-id.keyring On a centos8 client machine I have installed ceph-common, placed the Keyring file into /etc/ceph/, and run the command: rbd device map my-pool/my-image --id my-id All *seems* AOK. However - and here's my issue - when I try to create a partition on /dev/rbd0 and/or try to mount it, the client reports: fdisk: cannot open /dev/rbd0: Input/output error OR mount: /my-rbd-bloc-device: special device /dev/rbd0 does not exist (respectively). What am I doing wrong? Thanks in advance for the help Matthew J -- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.

4 4

How to know which client hold the lock of a file
by Norman.Kern 23 Mar '21

23 Mar '21

Hi, Anyone knows how to know which client hold lock of a file in Ceph fs? I met a dead lock problem that a client holding on get the lock, but I don't kown which client held it.

2 1

Re: Multisite RGW - Large omap objects related with bilogs
by Scheurer François 23 Mar '21

23 Mar '21

Dear All We have the same question here, if anyone can help ... Thank you! Cheers Francois ________________________________ From: ceph-users on behalf of P. O. <posdub(a)gmail.com> Sent: Friday, August 9, 2019 11:05 AM To: ceph-users(a)lists.ceph.com Subject: [ceph-users] Multisite RGW - Large omap objects related with bilogs Hi all, I have two ceph clusters in RGW multisite environment, with ~1500 bucketes ( 500M objects, 70TB ). Some of the buckets are very dynamic (objects are constantly changing). I have problems with large omap objects in bucket indexes, related with "dynamic buckets". For example: [root@rgw ~]# radosgw-admin bucket stats --bucket bucket_s3d33 |grep num_objects "num_objects": 564 In /var/log/ceph/ceph.log: cluster [WRN] Large omap object found. Object: 10:297646ca:::.dir.86a05ec8-9982-429b-9f94-28363610a95c.12546d0.17892:head Key count: 5307523 Size (bytes): 748792509 I found, this is because of bucket index logs: [root@rgw-1 ~]# rados -p default.rgw.buckets.index listomapkeys .dir.86a05ec8-9982-429b-9f94-28363610a95c.12546d0.17892 | wc -l 5307523 There are a lot of keys: �0_00013758656.71188336.4 �0_00013758657.71188337.5 �0_00013758658.71188338.4 �0_00013758659.71188339.5 �0_00013758660.71188342.4 �0_00013758661.71188343.5 �0_00013758662.71188344.4 [root@rgw-1 ~]# radosgw-admin bilog list --bucket bucket_s3d33 --max-entries 6000000 |grep op_id | wc -l 5307523 I have configured parameters in my ceph.conf: rgw sync log trim concurrent buckets = 32 rgw sync log trim max buckets = 64 rgw sync log trim interval = 1200 rgw sync log trim min cold buckets = 4 But from two weeks, the omap key count is still growing. How can I safely clean these bilogs (with no bucket damage and no replication damage)? I found two radosgw-admin parameters related with bilogs trimming: 1) radosgw-admin bilog trim --bucket=bucket_s3d33 --start-marker XXXX --end-marker YYYY I dont know what values should be in: --start-marker XXXX --end-marker YYYY. Is it safe to use "bilog trim" on bucket with replication in progress? If yes, should i run this on both sites? 2) radosgw-admin bilog autotrim Is this command safe? Can I use autotrim on selected bucket? Maybe there is some other way, to delete bilogs? Best regards, P.O. << ATT00001.txt (0.4KB) (0.4KB) >>

1 0

DocuBetter Meeting -- APAC 25 Mar 2021 0100 UTC
by John Zachary Dover 22 Mar '21

22 Mar '21

There will be a DocuBetter meeting on Thursday, 25 Mar 2021 at 0100 UTC. We will discuss the Google Season of Docs proposal (the Comprehensive Contribution Guide), the rewriting of the cephadm documentation and the new sectin of the Teuthology Guide. DocuBetter Meeting -- APAC 25 Mar 2021 0100 UTC https://bluejeans.com/908675367 https://pad.ceph.com/p/Ceph_Documentation

1 0

March 2021 Tech Talk and Code Walk-through
by Mike Perez 22 Mar '21

22 Mar '21

Hi everyone! I'm excited to announce two talks we have on the schedule for March 2021: Persistent Bucket Notifications By Yuval Lifshitz https://ceph.io/ceph-tech-talks/ The stream starts on March 25th at 17:00 UTC / 18:00 CET / 1:00 PM EST / 10:00 AM PST Persistent bucket notifications are going to be introduced in Ceph "Pacific." The idea behind them is to allow for reliable and asynchronous delivery of notifications from the RADOS gateway (RGW) to the endpoint configured at the topic. Regular notifications could also be considered reliable since the delivery to the endpoint is performed synchronously during the request. However, this reliability is only from the RGW perspective, meaning that the client will not get an ACK until an ACK is received from the endpoint but does not retry if the endpoint is down or disconnected. Also, note that, with regular notifications, if the endpoint sends back a NACK, the operation is still considered successful (since there is no way to rollback the RADOS operations that happened before the notification was tried). When the endpoint is down but failed to push, the notification is only timeout base; using regular notifications will slow down the operation of the RGW and may bring it to a complete halt. With persistent notifications, we allow the RGW to retry sending notifications even if the endpoint is down or a network disconnect with it during the operation (notifications are retried if not successfully delivered to the endpoint). Also, the operation is a-synchronous, so during the operation, the notifications are just pushed into a queue (see below), and the actual sending to the endpoint is happening a-synchronously. The queuing operation is done in 2 phases (reserve and commit or abort) to guarantee the queuing operation's atomicity with the other operations. In case you missed the last Tech Talk, see Sage Weil's presentation on What's new in Pacific: https://www.youtube.com/watch?v=PVtn53MbxTc -------------- Samuel Just will be giving a code walkthrough on RADOS Snapshots. https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs The stream starts on March 23rd at 17:00 UTC / 18:00 CET / 1:00 PM EST / 10:00 AM PST In case you missed it, watch Part 2 of LibRBD I/O Flow by Jason Dillaman: https://www.youtube.com/watch?v=nVjYVmqNClM All live streams will be recorded. -- Mike Perez (thingee)

1 0

how to disable write-back mode in ceph octopus
by 无名万剑归宗 22 Mar '21

22 Mar '21

I tried cache tier in write-back mode in my cluster, but because my ssd drive is home used, can not satisfy the needs of IOPS. Now I want disable write-back mode , I founded office documents,but the doc was outdated https://docs.ceph.com/en/latest/rados/operations/cache-tiering/?highlight=c… ）. root@e9000-22:~# ceph osd tier cache-mode cache proxy > Invalid command: proxy not in writeback|readproxy|readonly|none > osd tier cache-mode <pool> writeback|readproxy|readonly|none > [--yes-i-really-mean-it] : specify the caching mode for cache tier <pool> > Error EINVAL: invalid command Anyone can tell my how to disable this mode.

1 0

Question about migrating from iSCSI to RBD
by Justin Goetz 22 Mar '21

22 Mar '21

Hello! I was hoping to inquire if anyone here has attempted similar operations, and if they ran into any issues. To give a brief overview of my situation, I have a standard octopus cluster running 15.2.2, with ceph-iscsi installed via ansible. The original scope of a project we were working on changed, and we no longer need the iSCSI overhead added to the project (the machine using CEPH is Linux, so we would like to use native RBD block devices instead). Ideally we would create some new pools and migrate the data from the iSCSI pools over to the new pools, however, due to the massive amount of data (close to 200 TB), we lack the physical resources necessary to copy the files. Digging a bit on the backend of the pools utilized by ceph-iscsi, it appears that the iSCSI utility uses standard RBD images on the actual backend: ~]# rbd info iscsi/pool-name rbd image 'pool-name': size 200 TiB in 52428800 objects order 22 (4 MiB objects) snapshot_count: 0 id: 137b45a37ad84a block_name_prefix: rbd_data.137b45a37ad84a format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: object map invalid, fast diff invalid create_timestamp: Thu Nov 12 16:14:31 2020 access_timestamp: Tue Mar 16 16:13:41 2021 modify_timestamp: Tue Mar 16 16:15:36 2021 And I can also see that, like a standard rbd image, our 1st iSCSI gateway currently holds the lock on the image: ]# rbd lock ls --pool iscsi pool-name There is 1 exclusive lock on this image. Locker ID Address client.3618592 auto 259361792 10.101.12.61:0/1613659642 Theoretically speaking, would I be able to simply stop & disable the tcmu-runner processes on all iSCSI gateways in our cluster, which would release the lock on the RBD image, then create another user with rwx permissions to the iscsi pool? Would this work, or am I missing something that would come back to bite me later on? Looking for any advice on this topic. Thanks in advance for reading! -- Justin Goetz Systems Engineer, TeraSwitch Inc. jgoetz(a)teraswitch.com 412-945-7045 (NOC) | 412-459-7945 (Direct)

2 2

How to sizing nfs-ganesha.
by Quang Lê 22 Mar '21

22 Mar '21

Hi guys, I'm using manila-openstack to provide a filesystem service using backend CEPHFS. My design use nfs-ganesha as the gateway for the VM in openstack mount to CephFS. I am having problems with sizing the ganesha-servers. Can anyone suggest me *what are the hardware requirements of the ganesha server are* or *what parameters are needed to consider when sizing a ganesha server* ? My simple topology in link: *https://i.imgur.com/xrYqxAh.png <https://i.imgur.com/xrYqxAh.png>* Thank you guys.

2 1

Incomplete pg , any chance to to make it survive or data loss :( ?
by Szabo, Istvan (Agoda) 22 Mar '21

22 Mar '21

Hi, What can I do with this pg to make it work? We lost and don't have the osds 61,122 but we have the 32,33,70. I've exported the pg chunk from them, but they are very small and when I imported back to another osd that osd never started again so I had to remove that chunk (44.1aas2, 44.1aas3) to be able to start the osd. [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg incomplete pg 44.1aa is incomplete, acting [59,128,127,43] (reducing pool cephfs1-data01-pool min_size from 3 may help; search ceph.com/docs for 'incomplete') [WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100 [WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time pg 44.1aa not scrubbed since 2021-01-14T05:50:23.852626+0100 [WRN] SLOW_OPS: 96 slow ops, oldest one blocked for 228287 sec, osd.59 has slow ops This is the pg query and pg map important parts: "probing_osds": [ "29(3)", "34(3)", "43(3)", "56(1)", "59(0)", "72(2)", "73(2)", "74(2)", "127(2)", "128(1)", "131(2)" ], "down_osds_we_would_probe": [ 32, 33, 61, 70, 122 ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { "detail": "peering_blocked_by_history_les_bound" osdmap e5666778 pg 44.1aa (44.1aa) -> up [59,128,127,43] acting [59,128,127,43] Internal ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 2

Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data
by 特木勒 22 Mar '21

22 Mar '21

Hi Istvan: Any update for directional sync? I am trying to upgrade Ceph to 15.2.10. Also I have some issues that RGW may crash after I ran data sync init. :( Thanks

1 0

ceph orch daemon add , separate db
by Philip Brown 19 Mar '21

19 Mar '21

I was having difficulty doing this myself, and I came across this semi-recent thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/T4R76XJN2NE… " I've tried adding OSDs with ceph orch daemon add ... but it's pretty limited. ...you can't [have] a separate db device. " Has this been fixed yet? Is it GOING to be fixed? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 1

howto:: emergency shutdown procedure and maintenance
by Adrian Sevcenco 19 Mar '21

19 Mar '21

Hi! What steps/procedures are required for emergency shutdown and for machine maintenance? Thanks a lot! Adrian

2 2

high number of kernel clients per osd slow down
by Andrej Filipcic 19 Mar '21

19 Mar '21

Hi, I am testing 15.2.10 on a large cluster (RH8). cephfs pool (size=1) with 122 nvme OSDs works fine till the number of clients is relatively low. Writing from 400 kernel clients (ior benchmark), 8 streams each, causes issues. Writes are initially fast at 100GB/s but then they drop to <1GB/s after few minutes, while the OSDs use 300% cpu each. My guess is that the OSDs are overloaded with requests from too many clients, since it does not happen till there are ~3-4 streams OSD. The OSDs log do not show anything problematic. tried to increase osd_op_num_threads_per_shard_ssd, did not help. Restarting OSDs recovers the situation for few minutes. Writing to HDD pool with 1500 HDDs does not have any issues at all under same conditions. Any hints, settings to improve this? Cheers, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

2 2

March Ceph Science Virtual User Group Meeting
by Kevin Hrpcek 19 Mar '21

19 Mar '21

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday March 24th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20210324 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: March 24th, 2021 15:00 UTC 4pm Central European 10am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0

Importance of bluefs fix in Octopus 15.2.10 ?
by Chris Palmer 19 Mar '21

19 Mar '21

When looking over the changelog for 15.2.10 I noticed some bluefs changes. One in particular caught my eye, and it was called out as a notable change: os/bluestore: fix huge reads/writes at BlueFS (pr#39701 <https://github.com/ceph/ceph/pull/39701>, Jianpeng Ma, Igor Fedotov) It wasn't obvious to me how important that one was for data integrity. We'll roll 15.2.10 onto our clusters shortly, but would appreciate an opinion on whether any delay in installing this would increase the risk of anything nasty (e.g. silent data corruption). If that was the case we will push the release through rather more urgently! Thanks, Chris

2 1

Re: v15.2.10 Octopus released
by David Galloway 18 Mar '21

18 Mar '21

Terribly sorry for the mistake. There was a bug in the script I use to sync packages to download.ceph.com that wasn't listing directories in the desired order. That meant the download.ceph.com/{rpm,deb}-octopus symlinks still pointed to 15.2.9. This is fixed. I'm re-running the container jobs to get those pushed too. On 3/18/21 10:45 AM, David Orman wrote: > Hi David, > > The "For Packages" link in your email/the blog posts do not appear to > work. Additionally, we browsed the repo, and it doesn't appear the > packages are uploaded, at least for debian-octopus: > http://download.ceph.com/debian-octopus/pool/main/c/ceph/. We only use > the release packages for cephadm bootstrapping, so it's not a > deal-breaker for us, just wanted to give you a head's up. > > Cheers, > David Orman > > On Thu, Mar 18, 2021 at 9:11 AM David Galloway <dgallowa(a)redhat.com> wrote: >> >> We're happy to announce the 10th backport release in the Octopus series. >> We recommend users to update to this release. For a detailed release >> notes with links & changelog please refer to the official blog entry at >> https://ceph.io/releases/v15-2-10-octopus-released >> >> Notable Changes >> --------------- >> >> * The containers include an updated tcmalloc that avoids crashes seen on >> 15.2.9. See `issue#49618 <https://tracker.ceph.com/issues/49618>`_ for >> details. >> >> * RADOS: BlueStore handling of huge(>4GB) writes from RocksDB to BlueFS >> has been fixed. >> >> * When upgrading from a previous cephadm release, systemctl may hang >> when trying to start or restart the monitoring containers. (This is >> caused by a change in the systemd unit to use `type=forking`.) After the >> upgrade, please run:: >> >> ceph orch redeploy nfs >> ceph orch redeploy iscsi >> ceph orch redeploy node-exporter >> ceph orch redeploy prometheus >> ceph orch redeploy grafana >> ceph orch redeploy alertmanager >> >> >> Getting Ceph >> ------------ >> * Git at git://github.com/ceph/ceph.git >> * Tarball at http://download.ceph.com/tarballs/ceph-15.2.10.tar.gz >> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ >> * Release git sha1: 27917a557cca91e4da407489bbaa64ad4352cc02 >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Email alerts from Ceph
by Andrew Walker-Brown 18 Mar '21

18 Mar '21

Hi all, How have folks implemented getting email or snmp alerts out of Ceph? Getting things like osd/pool nearly full or osd/daemon failures etc. Kind regards Andrew Sent from my iPhone

5 4

v15.2.10 Octopus released
by Dimitri Savineau 18 Mar '21

18 Mar '21

Hi, https://download.ceph.com/rpm-octopus/ symlink isn't updated to the new release [1] and the content is still 15.2.9 [2] As a consequence, the new Octopus container images 15.2.10 can't be built. [1] https://download.ceph.com/rpm-15.2.10/ [2] https://download.ceph.com/rpm-15.2.9/ Regards, Dimitri

1 0

v15.2.10 Octopus released
by David Galloway 18 Mar '21

18 Mar '21

We're happy to announce the 10th backport release in the Octopus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-10-octopus-released Notable Changes --------------- * The containers include an updated tcmalloc that avoids crashes seen on 15.2.9. See `issue#49618 <https://tracker.ceph.com/issues/49618>`_ for details. * RADOS: BlueStore handling of huge(>4GB) writes from RocksDB to BlueFS has been fixed. * When upgrading from a previous cephadm release, systemctl may hang when trying to start or restart the monitoring containers. (This is caused by a change in the systemd unit to use `type=forking`.) After the upgrade, please run:: ceph orch redeploy nfs ceph orch redeploy iscsi ceph orch redeploy node-exporter ceph orch redeploy prometheus ceph orch redeploy grafana ceph orch redeploy alertmanager Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 27917a557cca91e4da407489bbaa64ad4352cc02

3 2

MON slow ops and growing MON store
by Janek Bevendorff 18 Mar '21

18 Mar '21

Hi, All of a sudden, we are experiencing very concerning MON behaviour. We have five MONs and all of them have thousands up to tens of thousands of slow ops, the oldest one blocking basically indefinitely (at least the timer keeps creeping up). Additionally, the MON stores keep inflating heavily. Under normal circumstances we have about 450-550MB there. Right now its 27GB and growing (rapidly). I tried restarting all MONs, I disabled auto-scaling (just in case) and checked the system load and hardware. I also restarted the MGR and MDS daemons, but to no avail. Is there any way I can debug this properly? I can’t seem to find how I can actually view what ops are causing this and what client (if any) may be responsible for it. Thanks Janek

4 13

PG export import
by Szabo, Istvan (Agoda) 18 Mar '21

18 Mar '21

Hi, I’ve tried to save some pg from a dead osd, I made this: Picked on the same server an osd which is not really used and stopped that osd and import the exported one from the dead one. root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-33 --no-mon-config --pgid 44.c0s0 --op export --file ./pg44c0s0 Exporting 44.c0s0 info 44.c0s0( empty local-lis/les=0/0 n=0 ec=192123/175799 lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493) Export successful root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-34 --no-mon-config --op import --file ./pg44c0s0 get_pg_num_history pg_num_history pg_num_history(e5583546 pg_nums {20={173213=256},21={219434=64},22={220991=64},24={219240=32},25={1446965=128},42={175793=32},43={197388=64},44={192123=512}} deleted_pools ) Importing pgid 44.c0s0 write_pg epoch 4865498 info 44.c0s0( empty local-lis/les=0/0 n=0 ec=192123/175799 lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493) Import successful Started back 34 and it says the osd is running but in the cluster map it is down :/ root@server:~# systemctl status ceph-osd@34 -l ● ceph-osd(a)34.service<mailto:ceph-osd@34.service> - Ceph object storage daemon osd.34 Loaded: loaded (/lib/systemd/system/ceph-osd@.service<mailto:/lib/systemd/system/ceph-osd@.service>; enabled-runtime; vendor preset: enabled) Active: active (running) since Thu 2021-03-18 10:38:00 CET; 8min ago Process: 45388 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 34 (code=exited, sta> Main PID: 45392 (ceph-osd) Tasks: 60 Memory: 856.2M CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)34.service └─45392 /usr/bin/ceph-osd -f --cluster ceph --id 34 --setuser ceph --setgroup ceph Mar 18 10:38:00 server systemd[1]: Starting Ceph object storage daemon osd.34... Mar 18 10:38:00 server systemd[1]: Started Ceph object storage daemon osd.34. Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.817+0100 7f41738d5dc0 -1 osd.34 5583546 log_to_mon> Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.825+0100 7f41738d5dc0 -1 osd.34 5583546 mon_cmd_ma> Any idea? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Same data for two buildings
by Denis Morejon Lopez 18 Mar '21

18 Mar '21

⁣I have a ceph cluster with 5 nodes. 3 in one building, and 2 in the other one. I put this information in the CRUSH. So that ceph were able to put one copy of objects in the nodes of one building and the other copy to the nodes of the other building. I mean, I setup replicas=2 in order to put the same information in both locations. But I know ceph cluster needs half +1 up nodes to keep one part of cluster working. I need at least a manual procedure to recover one of the two buildings if the other get down, or even if the link between them get down. I do not need 100% up, just something to block and unblock some nodes, and starts the two nodes if the building down were the one with 3 nodes.

2 1

Telemetry ident use?
by Matthew Vernon 17 Mar '21

17 Mar '21

Hi, What use is made of the ident data in the telemetry module? It's disabled by default, and the docs don't seem to say what it's used for... Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

1 0

Erasure-coded Block Device Image Creation With qemu-img - Help
by duluxoz 17 Mar '21

17 Mar '21

Hi Guys, So, new issue (I'm gonna get the hang of this if it kills me :-) ). I have a working/healthy Ceph (Octopus) Cluster (with qemu-img, libvert, etc, installed), and an erasure-coded pool called "my_pool". I now need to create a "my_data" image within the "my_pool" pool. As this is for a KVM host / block device (hence the qemu-img et.al.) I'm attempting to use qemu-img, so the command I am using is: ``` qemu-img create -f rbd rbd:my_pool/my_data 1T ``` The error message I received was: ``` qemu-img: rbd:my_pool/my_data: error rbd create: Operation not supported ``` So, I tried the 'raw' rbd command: ``` rbd create -s 1T my_pool/my_data ``` and got the error: ``` _add_image_to_directory: error adding image to directory: (95) Operation not supported rbd: create error: (95) Operation not supported ``` So I don't believe the issue is with the 'qemu-img' command - but I may be wrong. After doing some research I *think* I need to specify a replicated (as opposed to erasure-coded) pool for my_pool's metadata (eg 'my_pool_metadata'), and thus use the command: ``` rbd create -s 1T --data-pool my_pool my_pool_metadata/my_data ``` First Question: Is this correct? Second Question: What is the qemu-img equivalent command - is it: ``` qemu-img create -f rbd rbd:--data-pool my_pool my_pool_metadata/my_data 1T ``` or something similar? Thanks in advance Dulux-Oz

3 2

Quick quota question
by Andrew Walker-Brown 17 Mar '21

17 Mar '21

Hi all When setting a quota on a pool (or directory in Cephfs), is it the amount of client data written or the client data x number of replicas that counts toward the quota? Cheers A Sent from my iPhone

4 6

Diskless boot for Ceph nodes
by Stephen Smith6 17 Mar '21

17 Mar '21

6 6

Networking Idea/Question
by Dave Hall 17 Mar '21

17 Mar '21

Hello, If anybody out there has tried this or thought about it, I'd like to know... I've been thinking about ways to squeeze as much performance as possible from the NICs on a Ceph OSD node. The nodes in our cluster (6 x OSD, 3 x MGR/MON/MDS/RGW) currently have 2 x 10GB ports. Currently, one port is assigned to the front-side network, and one to the back-side network. However, there are times when the traffic on one side or the other is more intense and might benefit from a bit more bandwidth. The idea I had was to bond the two ports together, and to run the back-side network in a tagged VLAN on the combined 20GB LACP port. In order to keep the balance and prevent starvation from either side it would be necessary to apply some sort of a weighted fair queuing mechanism via the 'tc' command. The idea is that if the client side isn't using up the full 10GB/node, and there is a burst of re-balancing activity, the bandwidth consumed by the back-side traffic could swell to 15GB or more. Or vice versa. From what I have read and studied, these algorithms are fairly responsive to changes in load and would thus adjust rapidly if the demand from either side suddenly changed. Maybe this is a crazy idea, or maybe it's really cool. Your thoughts? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

7 11

Ceph Cluster Taking An Awful Long Time To Rebalance
by duluxoz 17 Mar '21

17 Mar '21

Hi Guys, Is the below "ceph -s" normal? This is a brand new cluster with (at the moment) a single Monitor and 7 OSDs (each 6 GiB) that has no data in it (yet), and yet its taking almost a day to "heal itself" from adding in the 2nd OSD. ~~~ cluster: id: [REDACTED] health: HEALTH_WARN Reduced data availability: 256 pgs inactive, 256 pgs incomplete Degraded data redundancy: 12 pgs undersized services: mon: 1 daemons, quorum [REDACTED] (age 22h) mgr: [REDACTED](active, since 22h) osd: 7 osds: 7 up (since 21h), 7 in (since 21h); 32 remapped pgs data: pools: 5 pools, 288 pgs objects: 7 objects, 0 B usage: 7.1 GiB used, 38 TiB / 38 TiB avail pgs: 88.889% pgs not active 6/21 objects misplaced (28.571%) 256 creating+incomplete 18 active+clean 12 active+undersized+remapped 2 active+clean+remapped progress: Rebalancing after osd.1 marked in (22h) [............................] PG autoscaler decreasing pool 1 PGs from 32 to 1 (19h) [............................] ~~~ Thanks in advance Matthew J -- Peregrine IT Signature *Matthew J BLACK* M.Inf.Tech.(Data Comms) MBA B.Sc. MACS (Snr), CP, IP3P When you want it done /right/ ‒ the first time! Phone: +61 4 0411 0089 Email: matthew(a)peregrineit.net <mailto:matthew@peregrineit.net> Web: www.peregrineit.net <http://www.peregrineit.net> View Matthew J BLACK's profile on LinkedIn <http://au.linkedin.com/in/mjblack> This Email is intended only for the addressee. Its use is limited to that intended by the author at the time and it is not to be distributed without the author’s consent. You must not use or disclose the contents of this Email, or add the sender’s Email address to any database, list or mailing list unless you are expressly authorised to do so. Unless otherwise stated, Peregrine I.T. Pty Ltd accepts no liability for the contents of this Email except where subsequently confirmed in writing. The opinions expressed in this Email are those of the author and do not necessarily represent the views of Peregrine I.T. Pty Ltd. This Email is confidential and may be subject to a claim of legal privilege. If you have received this Email in error, please notify the author and delete this message immediately.

3 6